[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260806 20:13:52.191753 29419 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.186.254:41559
I20260806 20:13:52.193336 29419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:13:52.194451 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:13:52.214989 29428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:13:52.215081 29426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:13:52.215721 29425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:13:52.218003 29419 server_base.cc:1034] running on GCE node
I20260806 20:13:53.361754 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:13:53.362006 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:13:53.362211 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047233362188 us; error 0 us; skew 500 ppm
I20260806 20:13:53.363080 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:13:53.369068 29419 webserver.cc:460] Webserver started at http://127.28.186.254:41843/ using document root <none> and password file <none>
I20260806 20:13:53.370050 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:13:53.370275 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:13:53.370738 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:13:53.377802 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/master-0-root/instance:
uuid: "aceb4a2bf355451fa5aa3543394be475"
format_stamp: "Formatted at 2026-08-06 20:13:53 on dist-test-slave-49h3"
I20260806 20:13:53.386962 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260806 20:13:53.392987 29434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:53.394479 29419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 20:13:53.394853 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/master-0-root
uuid: "aceb4a2bf355451fa5aa3543394be475"
format_stamp: "Formatted at 2026-08-06 20:13:53 on dist-test-slave-49h3"
I20260806 20:13:53.395220 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:13:53.432842 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:13:53.434541 29419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:13:53.435123 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:13:53.518458 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.254:41559
I20260806 20:13:53.518563 29487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.254:41559 every 8 connection(s)
I20260806 20:13:53.526060 29488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:53.552865 29488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475: Bootstrap starting.
I20260806 20:13:53.558903 29488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:53.561143 29488 log.cc:824] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475: Log is configured to *not* fsync() on all Append() calls
I20260806 20:13:53.566406 29488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475: No bootstrap required, opened a new log
I20260806 20:13:53.592465 29488 raft_consensus.cc:348] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aceb4a2bf355451fa5aa3543394be475" member_type: VOTER }
I20260806 20:13:53.593145 29488 raft_consensus.cc:374] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:53.593423 29488 raft_consensus.cc:729] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aceb4a2bf355451fa5aa3543394be475, State: Initialized, Role: FOLLOWER
I20260806 20:13:53.594345 29488 consensus_queue.cc:260] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aceb4a2bf355451fa5aa3543394be475" member_type: VOTER }
I20260806 20:13:53.594837 29488 raft_consensus.cc:388] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:13:53.595144 29488 raft_consensus.cc:482] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:13:53.595461 29488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:53.599687 29488 raft_consensus.cc:504] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aceb4a2bf355451fa5aa3543394be475" member_type: VOTER }
I20260806 20:13:53.600476 29488 leader_election.cc:302] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aceb4a2bf355451fa5aa3543394be475; no voters:
I20260806 20:13:53.602993 29488 leader_election.cc:288] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:13:53.603303 29491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:53.606638 29491 raft_consensus.cc:686] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 1 LEADER]: Becoming Leader. State: Replica: aceb4a2bf355451fa5aa3543394be475, State: Running, Role: LEADER
I20260806 20:13:53.607594 29491 consensus_queue.cc:237] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aceb4a2bf355451fa5aa3543394be475" member_type: VOTER }
I20260806 20:13:53.608628 29488 sys_catalog.cc:564] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:13:53.619175 29493 sys_catalog.cc:455] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aceb4a2bf355451fa5aa3543394be475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aceb4a2bf355451fa5aa3543394be475" member_type: VOTER } }
I20260806 20:13:53.620092 29493 sys_catalog.cc:458] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [sys.catalog]: This master's current role is: LEADER
I20260806 20:13:53.619881 29494 sys_catalog.cc:455] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aceb4a2bf355451fa5aa3543394be475. Latest consensus state: current_term: 1 leader_uuid: "aceb4a2bf355451fa5aa3543394be475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aceb4a2bf355451fa5aa3543394be475" member_type: VOTER } }
I20260806 20:13:53.620493 29494 sys_catalog.cc:458] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [sys.catalog]: This master's current role is: LEADER
I20260806 20:13:53.624715 29500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:13:53.637755 29500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:13:53.650063 29419 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 20:13:53.658352 29500 catalog_manager.cc:1347] Generated new cluster ID: 1a943ff52ea041e48d071abf4b4fa30c
I20260806 20:13:53.658681 29500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:13:53.684373 29500 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:13:53.687317 29500 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:13:53.711740 29500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475: Generated new TSK 0
I20260806 20:13:53.713032 29500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:13:53.783172 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:13:53.792233 29511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:13:53.793537 29512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:13:53.797727 29419 server_base.cc:1034] running on GCE node
W20260806 20:13:53.797932 29514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:13:53.798871 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:13:53.799055 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:13:53.799237 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047233799220 us; error 0 us; skew 500 ppm
I20260806 20:13:53.799819 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:13:53.804661 29419 webserver.cc:460] Webserver started at http://127.28.186.193:37351/ using document root <none> and password file <none>
I20260806 20:13:53.805377 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:13:53.805639 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:13:53.805907 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:13:53.807214 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-0-root/instance:
uuid: "a10e98dae4fb4f52b56f05afd3f86313"
format_stamp: "Formatted at 2026-08-06 20:13:53 on dist-test-slave-49h3"
I20260806 20:13:53.813582 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 20:13:53.818320 29519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:53.819175 29419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 20:13:53.819530 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-0-root
uuid: "a10e98dae4fb4f52b56f05afd3f86313"
format_stamp: "Formatted at 2026-08-06 20:13:53 on dist-test-slave-49h3"
I20260806 20:13:53.819923 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:13:53.850935 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:13:53.852839 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:13:53.856293 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:13:53.862579 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:13:53.862908 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:53.863216 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:13:53.863495 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:53.977131 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.193:39817
I20260806 20:13:53.977274 29586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.193:39817 every 8 connection(s)
I20260806 20:13:53.985209 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:13:53.995827 29592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:13:54.008379 29593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:13:54.019320 29587 heartbeater.cc:347] Connected to a master server at 127.28.186.254:41559
I20260806 20:13:54.019944 29587 heartbeater.cc:464] Registering TS with master...
I20260806 20:13:54.021417 29587 heartbeater.cc:511] Master 127.28.186.254:41559 requested a full tablet report, sending...
I20260806 20:13:54.027202 29452 ts_manager.cc:194] Registered new tserver with Master: a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817)
I20260806 20:13:54.031847 29452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43344
I20260806 20:13:54.044142 29419 server_base.cc:1034] running on GCE node
W20260806 20:13:54.044129 29595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:13:54.045409 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:13:54.045684 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:13:54.045855 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047234045843 us; error 0 us; skew 500 ppm
I20260806 20:13:54.046502 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:13:54.049496 29419 webserver.cc:460] Webserver started at http://127.28.186.194:46041/ using document root <none> and password file <none>
I20260806 20:13:54.050014 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:13:54.050213 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:13:54.050477 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:13:54.051564 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-1-root/instance:
uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
format_stamp: "Formatted at 2026-08-06 20:13:54 on dist-test-slave-49h3"
I20260806 20:13:54.057226 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:13:54.061417 29600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:54.062314 29419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:13:54.062590 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-1-root
uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
format_stamp: "Formatted at 2026-08-06 20:13:54 on dist-test-slave-49h3"
I20260806 20:13:54.062855 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:13:54.077013 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:13:54.078367 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:13:54.080821 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:13:54.084419 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:13:54.084626 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:54.084877 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:13:54.085018 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:54.198503 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.194:45689
I20260806 20:13:54.198637 29668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.194:45689 every 8 connection(s)
I20260806 20:13:54.205379 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:13:54.213093 29672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:13:54.219061 29673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:13:54.222129 29675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:13:54.223641 29669 heartbeater.cc:347] Connected to a master server at 127.28.186.254:41559
I20260806 20:13:54.224041 29669 heartbeater.cc:464] Registering TS with master...
I20260806 20:13:54.224902 29669 heartbeater.cc:511] Master 127.28.186.254:41559 requested a full tablet report, sending...
I20260806 20:13:54.227070 29452 ts_manager.cc:194] Registered new tserver with Master: d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:54.228545 29452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43348
I20260806 20:13:54.230784 29419 server_base.cc:1034] running on GCE node
I20260806 20:13:54.231711 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:13:54.231909 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:13:54.232069 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047234232053 us; error 0 us; skew 500 ppm
I20260806 20:13:54.232545 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:13:54.235266 29419 webserver.cc:460] Webserver started at http://127.28.186.195:46375/ using document root <none> and password file <none>
I20260806 20:13:54.235736 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:13:54.235917 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:13:54.236179 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:13:54.237244 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-2-root/instance:
uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
format_stamp: "Formatted at 2026-08-06 20:13:54 on dist-test-slave-49h3"
I20260806 20:13:54.241897 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 20:13:54.245404 29681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:54.246273 29419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:13:54.246557 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-2-root
uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
format_stamp: "Formatted at 2026-08-06 20:13:54 on dist-test-slave-49h3"
I20260806 20:13:54.246829 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786047231998201-29419-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:13:54.266956 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:13:54.268146 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:13:54.270879 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:13:54.274148 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:13:54.274363 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:54.274616 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:13:54.274765 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:13:54.388216 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.195:42515
I20260806 20:13:54.388386 29749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.195:42515 every 8 connection(s)
I20260806 20:13:54.409569 29750 heartbeater.cc:347] Connected to a master server at 127.28.186.254:41559
I20260806 20:13:54.409960 29750 heartbeater.cc:464] Registering TS with master...
I20260806 20:13:54.410878 29750 heartbeater.cc:511] Master 127.28.186.254:41559 requested a full tablet report, sending...
I20260806 20:13:54.412809 29452 ts_manager.cc:194] Registered new tserver with Master: 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:54.413638 29419 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018197714s
I20260806 20:13:54.414511 29452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43360
I20260806 20:13:54.417879 29419 test_util.cc:268] Using random seed: -2030721860
I20260806 20:13:54.478205 29452 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43368:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:13:54.481942 29452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:13:54.603518 29711 tablet_service.cc:1467] Processing CreateTablet for tablet d150978500404c0c84d2c1be4767b06f (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 20:13:54.611209 29706 tablet_service.cc:1467] Processing CreateTablet for tablet d133841d936c42e5a45fe986450328e3 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 20:13:54.613538 29707 tablet_service.cc:1467] Processing CreateTablet for tablet 3a8b884e76a547bf8ff5a7c371f629a1 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 20:13:54.611542 29710 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd4523c8fcd4e3c950fe41d0a908f19 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 20:13:54.614348 29705 tablet_service.cc:1467] Processing CreateTablet for tablet 2b08a45d83e140d7b6a07558f4da9a63 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 20:13:54.612293 29708 tablet_service.cc:1467] Processing CreateTablet for tablet b3ff2293355b476b85beb264fc24f12d (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 20:13:54.620491 29709 tablet_service.cc:1467] Processing CreateTablet for tablet f6e60d5891ab409ea2a360dd221d3d1e (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 20:13:54.619971 29703 tablet_service.cc:1467] Processing CreateTablet for tablet 3474e05293044f6cbbf938d0f54c9dd1 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 20:13:54.621018 29704 tablet_service.cc:1467] Processing CreateTablet for tablet 14689f97dadc4b0e95a25cedc9f172cc (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 20:13:54.623255 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d150978500404c0c84d2c1be4767b06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.628701 29710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd4523c8fcd4e3c950fe41d0a908f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.629758 29707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a8b884e76a547bf8ff5a7c371f629a1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:13:54.628980 29502 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:13:54.636257 29550 tablet_service.cc:1467] Processing CreateTablet for tablet d150978500404c0c84d2c1be4767b06f (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 20:13:54.637661 29549 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd4523c8fcd4e3c950fe41d0a908f19 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 20:13:54.646634 29547 tablet_service.cc:1467] Processing CreateTablet for tablet b3ff2293355b476b85beb264fc24f12d (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 20:13:54.649873 29545 tablet_service.cc:1467] Processing CreateTablet for tablet d133841d936c42e5a45fe986450328e3 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 20:13:54.651832 29548 tablet_service.cc:1467] Processing CreateTablet for tablet f6e60d5891ab409ea2a360dd221d3d1e (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 20:13:54.654019 29546 tablet_service.cc:1467] Processing CreateTablet for tablet 3a8b884e76a547bf8ff5a7c371f629a1 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 20:13:54.655176 29708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ff2293355b476b85beb264fc24f12d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.657755 29705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b08a45d83e140d7b6a07558f4da9a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.658437 29704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14689f97dadc4b0e95a25cedc9f172cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.659250 29709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e60d5891ab409ea2a360dd221d3d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.659988 29706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d133841d936c42e5a45fe986450328e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.660676 29703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3474e05293044f6cbbf938d0f54c9dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.671092 29544 tablet_service.cc:1467] Processing CreateTablet for tablet 2b08a45d83e140d7b6a07558f4da9a63 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 20:13:54.671329 29543 tablet_service.cc:1467] Processing CreateTablet for tablet 14689f97dadc4b0e95a25cedc9f172cc (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 20:13:54.654536 29550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d150978500404c0c84d2c1be4767b06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.674966 29543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14689f97dadc4b0e95a25cedc9f172cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.675103 29542 tablet_service.cc:1467] Processing CreateTablet for tablet 3474e05293044f6cbbf938d0f54c9dd1 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 20:13:54.675899 29544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b08a45d83e140d7b6a07558f4da9a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.676856 29542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3474e05293044f6cbbf938d0f54c9dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.693957 29546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a8b884e76a547bf8ff5a7c371f629a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.697535 29548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e60d5891ab409ea2a360dd221d3d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.698581 29545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d133841d936c42e5a45fe986450328e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.708067 29547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ff2293355b476b85beb264fc24f12d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.709864 29549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd4523c8fcd4e3c950fe41d0a908f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.766974 29626 tablet_service.cc:1467] Processing CreateTablet for tablet b3ff2293355b476b85beb264fc24f12d (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 20:13:54.763629 29629 tablet_service.cc:1467] Processing CreateTablet for tablet d150978500404c0c84d2c1be4767b06f (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 20:13:54.774202 29629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d150978500404c0c84d2c1be4767b06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.771900 29628 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd4523c8fcd4e3c950fe41d0a908f19 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 20:13:54.771163 29627 tablet_service.cc:1467] Processing CreateTablet for tablet f6e60d5891ab409ea2a360dd221d3d1e (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 20:13:54.780738 29626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ff2293355b476b85beb264fc24f12d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.771026 29625 tablet_service.cc:1467] Processing CreateTablet for tablet 3a8b884e76a547bf8ff5a7c371f629a1 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 20:13:54.779135 29624 tablet_service.cc:1467] Processing CreateTablet for tablet d133841d936c42e5a45fe986450328e3 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 20:13:54.797792 29624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d133841d936c42e5a45fe986450328e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.799831 29628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd4523c8fcd4e3c950fe41d0a908f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.809582 29625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a8b884e76a547bf8ff5a7c371f629a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.797789 29623 tablet_service.cc:1467] Processing CreateTablet for tablet 2b08a45d83e140d7b6a07558f4da9a63 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 20:13:54.813854 29627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e60d5891ab409ea2a360dd221d3d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.814862 29622 tablet_service.cc:1467] Processing CreateTablet for tablet 14689f97dadc4b0e95a25cedc9f172cc (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 20:13:54.818245 29622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14689f97dadc4b0e95a25cedc9f172cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.823050 29623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b08a45d83e140d7b6a07558f4da9a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.834573 29621 tablet_service.cc:1467] Processing CreateTablet for tablet 3474e05293044f6cbbf938d0f54c9dd1 (DEFAULT_TABLE table=test-workload [id=90f37428771e4c33bf0323a36b6d97f2]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 20:13:54.839811 29621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3474e05293044f6cbbf938d0f54c9dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:13:54.899155 29769 tablet_bootstrap.cc:492] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8: Bootstrap starting.
I20260806 20:13:54.895563 29768 tablet_bootstrap.cc:492] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313: Bootstrap starting.
I20260806 20:13:54.922221 29770 tablet_bootstrap.cc:492] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7: Bootstrap starting.
I20260806 20:13:54.956470 29770 tablet_bootstrap.cc:654] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:54.966187 29769 tablet_bootstrap.cc:654] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:54.986759 29768 tablet_bootstrap.cc:654] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:54.989068 29769 tablet_bootstrap.cc:492] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8: No bootstrap required, opened a new log
I20260806 20:13:54.992405 29770 tablet_bootstrap.cc:492] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7: No bootstrap required, opened a new log
I20260806 20:13:54.992818 29769 ts_tablet_manager.cc:1397] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent bootstrapping tablet: real 0.099s user 0.008s sys 0.008s
I20260806 20:13:54.992962 29770 ts_tablet_manager.cc:1397] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.003s
I20260806 20:13:54.996533 29770 raft_consensus.cc:348] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:54.997428 29770 raft_consensus.cc:374] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:54.997035 29769 raft_consensus.cc:348] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:54.997794 29770 raft_consensus.cc:729] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Initialized, Role: FOLLOWER
I20260806 20:13:54.998193 29769 raft_consensus.cc:374] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:54.998605 29769 raft_consensus.cc:729] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Initialized, Role: FOLLOWER
I20260806 20:13:54.998626 29770 consensus_queue.cc:260] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:54.999776 29769 consensus_queue.cc:260] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.004311 29768 tablet_bootstrap.cc:492] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313: No bootstrap required, opened a new log
I20260806 20:13:55.004774 29768 ts_tablet_manager.cc:1397] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313: Time spent bootstrapping tablet: real 0.136s user 0.009s sys 0.013s
I20260806 20:13:55.007767 29768 raft_consensus.cc:348] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:55.008527 29768 raft_consensus.cc:374] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.008831 29768 raft_consensus.cc:729] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.009938 29768 consensus_queue.cc:260] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:55.013617 29669 heartbeater.cc:503] Master 127.28.186.254:41559 was elected leader, sending a full tablet report...
I20260806 20:13:55.014633 29750 heartbeater.cc:503] Master 127.28.186.254:41559 was elected leader, sending a full tablet report...
I20260806 20:13:55.015285 29770 ts_tablet_manager.cc:1428] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent starting tablet: real 0.022s user 0.011s sys 0.009s
I20260806 20:13:55.016412 29770 tablet_bootstrap.cc:492] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7: Bootstrap starting.
I20260806 20:13:55.018595 29769 ts_tablet_manager.cc:1428] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent starting tablet: real 0.025s user 0.006s sys 0.002s
I20260806 20:13:55.019649 29769 tablet_bootstrap.cc:492] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8: Bootstrap starting.
I20260806 20:13:55.022635 29587 heartbeater.cc:503] Master 127.28.186.254:41559 was elected leader, sending a full tablet report...
I20260806 20:13:55.024591 29770 tablet_bootstrap.cc:654] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.027216 29774 raft_consensus.cc:482] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.029548 29768 ts_tablet_manager.cc:1428] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313: Time spent starting tablet: real 0.024s user 0.002s sys 0.007s
I20260806 20:13:55.027722 29774 raft_consensus.cc:504] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.031224 29768 tablet_bootstrap.cc:492] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313: Bootstrap starting.
I20260806 20:13:55.037827 29774 leader_election.cc:288] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.044715 29768 tablet_bootstrap.cc:654] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.041375 29769 tablet_bootstrap.cc:654] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.090260 29770 tablet_bootstrap.cc:492] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7: No bootstrap required, opened a new log
I20260806 20:13:55.090776 29770 ts_tablet_manager.cc:1397] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent bootstrapping tablet: real 0.075s user 0.025s sys 0.028s
I20260806 20:13:55.094194 29770 raft_consensus.cc:348] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.094977 29770 raft_consensus.cc:374] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.095300 29770 raft_consensus.cc:729] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.095973 29770 consensus_queue.cc:260] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.099462 29770 ts_tablet_manager.cc:1428] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 20:13:55.100616 29770 tablet_bootstrap.cc:492] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7: Bootstrap starting.
I20260806 20:13:55.107064 29770 tablet_bootstrap.cc:654] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.118804 29770 tablet_bootstrap.cc:492] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7: No bootstrap required, opened a new log
I20260806 20:13:55.119313 29770 ts_tablet_manager.cc:1397] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20260806 20:13:55.124385 29770 raft_consensus.cc:348] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.125190 29770 raft_consensus.cc:374] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.125511 29770 raft_consensus.cc:729] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.126284 29769 tablet_bootstrap.cc:492] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8: No bootstrap required, opened a new log
I20260806 20:13:55.126924 29769 ts_tablet_manager.cc:1397] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent bootstrapping tablet: real 0.107s user 0.035s sys 0.012s
I20260806 20:13:55.126334 29770 consensus_queue.cc:260] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.129006 29770 ts_tablet_manager.cc:1428] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260806 20:13:55.129989 29770 tablet_bootstrap.cc:492] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7: Bootstrap starting.
I20260806 20:13:55.129696 29769 raft_consensus.cc:348] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.130429 29769 raft_consensus.cc:374] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.130756 29769 raft_consensus.cc:729] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.131431 29769 consensus_queue.cc:260] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.133394 29778 raft_consensus.cc:482] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.133919 29778 raft_consensus.cc:504] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:55.136317 29778 leader_election.cc:288] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:55.136654 29770 tablet_bootstrap.cc:654] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.140899 29768 tablet_bootstrap.cc:492] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313: No bootstrap required, opened a new log
I20260806 20:13:55.141724 29768 ts_tablet_manager.cc:1397] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313: Time spent bootstrapping tablet: real 0.111s user 0.014s sys 0.035s
I20260806 20:13:55.144380 29768 raft_consensus.cc:348] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.145116 29768 raft_consensus.cc:374] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.145398 29768 raft_consensus.cc:729] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.146099 29768 consensus_queue.cc:260] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
W20260806 20:13:55.151206 29751 tablet.cc:2369] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:13:55.151291 29769 ts_tablet_manager.cc:1428] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260806 20:13:55.152451 29769 tablet_bootstrap.cc:492] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8: Bootstrap starting.
I20260806 20:13:55.156488 29768 ts_tablet_manager.cc:1428] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20260806 20:13:55.158413 29768 tablet_bootstrap.cc:492] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313: Bootstrap starting.
I20260806 20:13:55.163496 29770 tablet_bootstrap.cc:492] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7: No bootstrap required, opened a new log
I20260806 20:13:55.166810 29770 ts_tablet_manager.cc:1397] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.012s
I20260806 20:13:55.169981 29770 raft_consensus.cc:348] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.170708 29770 raft_consensus.cc:374] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.170980 29770 raft_consensus.cc:729] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.171923 29770 consensus_queue.cc:260] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.175239 29770 ts_tablet_manager.cc:1428] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260806 20:13:55.176321 29770 tablet_bootstrap.cc:492] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7: Bootstrap starting.
I20260806 20:13:55.178807 29769 tablet_bootstrap.cc:654] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.181746 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313" is_pre_election: true
W20260806 20:13:55.186079 29601 leader_election.cc:341] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:13:55.194064 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" is_pre_election: true
W20260806 20:13:55.197597 29522 leader_election.cc:341] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:13:55.199472 29770 tablet_bootstrap.cc:654] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.200552 29768 tablet_bootstrap.cc:654] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.204310 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" is_pre_election: true
I20260806 20:13:55.205736 29769 tablet_bootstrap.cc:492] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8: No bootstrap required, opened a new log
I20260806 20:13:55.206559 29769 ts_tablet_manager.cc:1397] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent bootstrapping tablet: real 0.054s user 0.016s sys 0.002s
W20260806 20:13:55.207647 29604 leader_election.cc:341] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:13:55.208310 29604 leader_election.cc:302] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5ad1e1b5e874ccea53daf19b90f22a7; no voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:13:55.209156 29774 raft_consensus.cc:2726] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:13:55.215902 29769 raft_consensus.cc:348] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:55.216779 29769 raft_consensus.cc:374] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.217088 29769 raft_consensus.cc:729] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.218322 29770 tablet_bootstrap.cc:492] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7: No bootstrap required, opened a new log
I20260806 20:13:55.218824 29770 ts_tablet_manager.cc:1397] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.003s
I20260806 20:13:55.218871 29774 raft_consensus.cc:482] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.219384 29774 raft_consensus.cc:504] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.219813 29769 consensus_queue.cc:260] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:55.222683 29769 ts_tablet_manager.cc:1428] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260806 20:13:55.223564 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" is_pre_election: true
I20260806 20:13:55.222858 29770 raft_consensus.cc:348] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:55.224084 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313" is_pre_election: true
I20260806 20:13:55.224982 29769 tablet_bootstrap.cc:492] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8: Bootstrap starting.
I20260806 20:13:55.225157 29770 raft_consensus.cc:374] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.226274 29770 raft_consensus.cc:729] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.229765 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" is_pre_election: true
I20260806 20:13:55.225255 29721 raft_consensus.cc:2454] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 0.
W20260806 20:13:55.235850 29601 leader_election.cc:341] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:13:55.235347 29770 consensus_queue.cc:260] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:55.236527 29720 raft_consensus.cc:2454] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 0.
I20260806 20:13:55.236725 29604 leader_election.cc:302] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters: a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:13:55.239099 29524 leader_election.cc:302] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters: d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:13:55.239626 29793 raft_consensus.cc:2781] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:55.239956 29778 raft_consensus.cc:2781] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:55.239969 29793 raft_consensus.cc:482] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:55.240372 29778 raft_consensus.cc:482] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:55.240562 29793 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.240749 29778 raft_consensus.cc:3037] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.224534 29774 leader_election.cc:288] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.247670 29778 raft_consensus.cc:504] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:55.248714 29793 raft_consensus.cc:504] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.250455 29778 leader_election.cc:288] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 election: Requested vote from peers 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:55.251266 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:55.251518 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:55.251879 29721 raft_consensus.cc:3037] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.252116 29641 raft_consensus.cc:3037] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.251873 29769 tablet_bootstrap.cc:654] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.252868 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
W20260806 20:13:55.254222 29601 leader_election.cc:341] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:13:55.256237 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:55.256820 29720 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.251526 29768 tablet_bootstrap.cc:492] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313: No bootstrap required, opened a new log
I20260806 20:13:55.255920 29793 leader_election.cc:288] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.258208 29768 ts_tablet_manager.cc:1397] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313: Time spent bootstrapping tablet: real 0.101s user 0.021s sys 0.000s
I20260806 20:13:55.260039 29770 ts_tablet_manager.cc:1428] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent starting tablet: real 0.041s user 0.007s sys 0.006s
I20260806 20:13:55.260325 29721 raft_consensus.cc:2454] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 1.
I20260806 20:13:55.261029 29770 tablet_bootstrap.cc:492] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7: Bootstrap starting.
I20260806 20:13:55.262076 29524 leader_election.cc:302] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:13:55.262092 29768 raft_consensus.cc:348] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.263494 29768 raft_consensus.cc:374] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.263784 29768 raft_consensus.cc:729] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.264457 29768 consensus_queue.cc:260] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.266657 29778 raft_consensus.cc:2781] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:55.268138 29770 tablet_bootstrap.cc:654] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.269119 29720 raft_consensus.cc:2454] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 1.
I20260806 20:13:55.270701 29604 leader_election.cc:302] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters: a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:13:55.269119 29641 raft_consensus.cc:2454] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 1.
I20260806 20:13:55.271471 29793 raft_consensus.cc:2781] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:55.272064 29793 raft_consensus.cc:686] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:13:55.272930 29793 consensus_queue.cc:237] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.275643 29769 tablet_bootstrap.cc:492] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8: No bootstrap required, opened a new log
I20260806 20:13:55.276125 29769 ts_tablet_manager.cc:1397] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.009s
I20260806 20:13:55.276515 29768 ts_tablet_manager.cc:1428] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260806 20:13:55.282074 29768 tablet_bootstrap.cc:492] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313: Bootstrap starting.
I20260806 20:13:55.283318 29769 raft_consensus.cc:348] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.283924 29778 raft_consensus.cc:686] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Becoming Leader. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:13:55.284576 29769 raft_consensus.cc:374] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.285799 29769 raft_consensus.cc:729] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.285337 29778 consensus_queue.cc:237] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:55.287613 29769 consensus_queue.cc:260] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.292889 29769 ts_tablet_manager.cc:1428] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent starting tablet: real 0.016s user 0.008s sys 0.004s
I20260806 20:13:55.301255 29770 tablet_bootstrap.cc:492] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7: No bootstrap required, opened a new log
I20260806 20:13:55.301801 29770 ts_tablet_manager.cc:1397] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.019s
I20260806 20:13:55.303838 29769 tablet_bootstrap.cc:492] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8: Bootstrap starting.
I20260806 20:13:55.304829 29770 raft_consensus.cc:348] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.305704 29770 raft_consensus.cc:374] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.306001 29770 raft_consensus.cc:729] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.306761 29770 consensus_queue.cc:260] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.309238 29770 ts_tablet_manager.cc:1428] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260806 20:13:55.310158 29770 tablet_bootstrap.cc:492] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7: Bootstrap starting.
I20260806 20:13:55.311801 29769 tablet_bootstrap.cc:654] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.311344 29452 catalog_manager.cc:5495] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:55.297515 29768 tablet_bootstrap.cc:654] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.314869 29450 catalog_manager.cc:5495] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 reported cstate change: term changed from 0 to 1, leader changed from <none> to a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:55.316398 29770 tablet_bootstrap.cc:654] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.324451 29769 tablet_bootstrap.cc:492] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8: No bootstrap required, opened a new log
I20260806 20:13:55.324935 29769 ts_tablet_manager.cc:1397] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.008s
I20260806 20:13:55.327618 29769 raft_consensus.cc:348] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.328354 29769 raft_consensus.cc:374] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.328656 29769 raft_consensus.cc:729] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.329401 29769 consensus_queue.cc:260] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.334998 29769 ts_tablet_manager.cc:1428] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260806 20:13:55.335641 29768 tablet_bootstrap.cc:492] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313: No bootstrap required, opened a new log
I20260806 20:13:55.335956 29769 tablet_bootstrap.cc:492] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8: Bootstrap starting.
I20260806 20:13:55.336150 29768 ts_tablet_manager.cc:1397] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.002s
I20260806 20:13:55.339259 29768 raft_consensus.cc:348] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.340167 29768 raft_consensus.cc:374] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.340476 29768 raft_consensus.cc:729] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.341159 29768 consensus_queue.cc:260] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.344096 29768 ts_tablet_manager.cc:1428] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20260806 20:13:55.345196 29768 tablet_bootstrap.cc:492] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313: Bootstrap starting.
I20260806 20:13:55.345721 29770 tablet_bootstrap.cc:492] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7: No bootstrap required, opened a new log
I20260806 20:13:55.346232 29770 ts_tablet_manager.cc:1397] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.004s
I20260806 20:13:55.347177 29778 raft_consensus.cc:482] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.347695 29778 raft_consensus.cc:504] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.348904 29770 raft_consensus.cc:348] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.349781 29778 leader_election.cc:288] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.350021 29770 raft_consensus.cc:374] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.350302 29770 raft_consensus.cc:729] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.350835 29770 consensus_queue.cc:260] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.351265 29768 tablet_bootstrap.cc:654] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.351505 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" is_pre_election: true
I20260806 20:13:55.352283 29641 raft_consensus.cc:2454] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 0.
I20260806 20:13:55.352568 29770 ts_tablet_manager.cc:1428] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260806 20:13:55.353304 29770 tablet_bootstrap.cc:492] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7: Bootstrap starting.
I20260806 20:13:55.353991 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" is_pre_election: true
I20260806 20:13:55.354657 29720 raft_consensus.cc:2454] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 0.
I20260806 20:13:55.356268 29522 leader_election.cc:302] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:55.357944 29770 tablet_bootstrap.cc:654] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.358647 29795 raft_consensus.cc:2781] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:55.358923 29778 raft_consensus.cc:482] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.359017 29795 raft_consensus.cc:482] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:55.359509 29778 raft_consensus.cc:504] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.359508 29795 raft_consensus.cc:3037] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.362280 29769 tablet_bootstrap.cc:654] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.366756 29778 leader_election.cc:288] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.368712 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" is_pre_election: true
W20260806 20:13:55.376850 29524 leader_election.cc:341] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:13:55.376405 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" is_pre_election: true
I20260806 20:13:55.380873 29641 raft_consensus.cc:2454] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 0.
I20260806 20:13:55.383603 29522 leader_election.cc:302] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters: 96cbcc4fe5f841d798d8b6131fafd3f8
I20260806 20:13:55.384752 29778 raft_consensus.cc:2781] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:55.385368 29778 raft_consensus.cc:482] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:55.385747 29778 raft_consensus.cc:3037] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.386888 29795 raft_consensus.cc:504] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.393966 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:55.394604 29641 raft_consensus.cc:3037] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.400545 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:55.401399 29720 raft_consensus.cc:3037] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.404474 29768 tablet_bootstrap.cc:492] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313: No bootstrap required, opened a new log
I20260806 20:13:55.404937 29768 ts_tablet_manager.cc:1397] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313: Time spent bootstrapping tablet: real 0.060s user 0.029s sys 0.021s
I20260806 20:13:55.410120 29777 raft_consensus.cc:482] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.410626 29777 raft_consensus.cc:504] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.402094 29795 leader_election.cc:288] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 election: Requested vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.415063 29777 leader_election.cc:288] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:55.420048 29641 raft_consensus.cc:2454] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 1.
I20260806 20:13:55.412102 29778 raft_consensus.cc:504] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.426061 29522 leader_election.cc:302] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:55.417793 29720 raft_consensus.cc:2454] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 1.
I20260806 20:13:55.431028 29778 leader_election.cc:288] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 election: Requested vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.432281 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:55.432885 29770 tablet_bootstrap.cc:492] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7: No bootstrap required, opened a new log
I20260806 20:13:55.434855 29770 ts_tablet_manager.cc:1397] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent bootstrapping tablet: real 0.082s user 0.044s sys 0.024s
I20260806 20:13:55.436805 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:55.433799 29795 raft_consensus.cc:2781] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:55.434535 29641 raft_consensus.cc:3037] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.438935 29795 raft_consensus.cc:686] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Becoming Leader. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
W20260806 20:13:55.438704 29524 leader_election.cc:341] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:13:55.447278 29768 raft_consensus.cc:348] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.448056 29768 raft_consensus.cc:374] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.448948 29768 raft_consensus.cc:729] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.451354 29641 raft_consensus.cc:2454] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 1.
I20260806 20:13:55.449805 29768 consensus_queue.cc:260] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.449532 29795 consensus_queue.cc:237] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.449407 29770 raft_consensus.cc:348] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.452370 29522 leader_election.cc:302] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters: 96cbcc4fe5f841d798d8b6131fafd3f8
I20260806 20:13:55.452857 29770 raft_consensus.cc:374] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.453120 29770 raft_consensus.cc:729] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.453696 29796 raft_consensus.cc:2781] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:55.453753 29770 consensus_queue.cc:260] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.458961 29768 ts_tablet_manager.cc:1428] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313: Time spent starting tablet: real 0.054s user 0.010s sys 0.002s
I20260806 20:13:55.459180 29769 tablet_bootstrap.cc:492] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8: No bootstrap required, opened a new log
I20260806 20:13:55.459681 29769 ts_tablet_manager.cc:1397] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent bootstrapping tablet: real 0.124s user 0.014s sys 0.007s
I20260806 20:13:55.457518 29770 ts_tablet_manager.cc:1428] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent starting tablet: real 0.022s user 0.003s sys 0.004s
I20260806 20:13:55.463018 29769 raft_consensus.cc:348] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.465268 29770 tablet_bootstrap.cc:492] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7: Bootstrap starting.
I20260806 20:13:55.464605 29769 raft_consensus.cc:374] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.465752 29769 raft_consensus.cc:729] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.465255 29768 tablet_bootstrap.cc:492] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313: Bootstrap starting.
I20260806 20:13:55.469236 29793 raft_consensus.cc:482] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.469909 29793 raft_consensus.cc:504] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.472450 29793 leader_election.cc:288] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.474149 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3474e05293044f6cbbf938d0f54c9dd1" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" is_pre_election: true
I20260806 20:13:55.475453 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3474e05293044f6cbbf938d0f54c9dd1" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313" is_pre_election: true
W20260806 20:13:55.476933 29604 leader_election.cc:341] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:13:55.468083 29769 consensus_queue.cc:260] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
W20260806 20:13:55.478200 29601 leader_election.cc:341] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:13:55.479063 29770 tablet_bootstrap.cc:654] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.478798 29796 raft_consensus.cc:686] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Becoming Leader. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:13:55.478684 29601 leader_election.cc:302] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5ad1e1b5e874ccea53daf19b90f22a7; no voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:13:55.482218 29793 raft_consensus.cc:2726] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:13:55.485795 29769 ts_tablet_manager.cc:1428] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent starting tablet: real 0.026s user 0.008s sys 0.001s
I20260806 20:13:55.481406 29796 consensus_queue.cc:237] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.488806 29769 tablet_bootstrap.cc:492] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8: Bootstrap starting.
I20260806 20:13:55.494676 29450 catalog_manager.cc:5495] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 reported cstate change: term changed from 0 to 1, leader changed from <none> to a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:55.501571 29815 raft_consensus.cc:482] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.512007 29769 tablet_bootstrap.cc:654] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.514178 29768 tablet_bootstrap.cc:654] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.508262 29815 raft_consensus.cc:504] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.518229 29770 tablet_bootstrap.cc:492] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7: No bootstrap required, opened a new log
I20260806 20:13:55.518587 29770 ts_tablet_manager.cc:1397] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.009s
I20260806 20:13:55.519486 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" is_pre_election: true
I20260806 20:13:55.520318 29641 raft_consensus.cc:2454] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 0.
I20260806 20:13:55.521090 29815 leader_election.cc:288] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.522209 29684 leader_election.cc:302] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:55.522346 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" is_pre_election: true
I20260806 20:13:55.523031 29641 raft_consensus.cc:2454] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 0.
I20260806 20:13:55.523113 29777 raft_consensus.cc:2781] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:55.522447 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" is_pre_election: true
I20260806 20:13:55.523490 29777 raft_consensus.cc:482] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:55.523828 29777 raft_consensus.cc:3037] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.525202 29522 leader_election.cc:302] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:55.526113 29815 raft_consensus.cc:2781] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:55.526285 29770 raft_consensus.cc:348] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.527110 29770 raft_consensus.cc:374] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.527413 29770 raft_consensus.cc:729] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.525758 29452 catalog_manager.cc:5495] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 reported cstate change: term changed from 0 to 1, leader changed from <none> to a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:55.528227 29770 consensus_queue.cc:260] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.526490 29815 raft_consensus.cc:482] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:55.529529 29815 raft_consensus.cc:3037] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.530833 29770 ts_tablet_manager.cc:1428] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260806 20:13:55.531657 29777 raft_consensus.cc:504] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.534039 29777 leader_election.cc:288] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:55.533365 29720 raft_consensus.cc:2379] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a10e98dae4fb4f52b56f05afd3f86313 in current term 1: Already voted for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in this term.
I20260806 20:13:55.536993 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:55.537591 29641 raft_consensus.cc:3037] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.538630 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313" is_pre_election: true
I20260806 20:13:55.541770 29815 raft_consensus.cc:504] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.542078 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:55.543107 29560 raft_consensus.cc:2379] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in current term 1: Already voted for candidate a10e98dae4fb4f52b56f05afd3f86313 in this term.
I20260806 20:13:55.543967 29815 leader_election.cc:288] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 election: Requested vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.546020 29640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:55.554304 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:55.555285 29720 raft_consensus.cc:2379] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a10e98dae4fb4f52b56f05afd3f86313 in current term 1: Already voted for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in this term.
I20260806 20:13:55.556541 29524 leader_election.cc:302] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313; no voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:13:55.557175 29815 raft_consensus.cc:2726] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:13:55.558558 29641 raft_consensus.cc:2454] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 1.
I20260806 20:13:55.559559 29684 leader_election.cc:302] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters: a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:13:55.560385 29777 raft_consensus.cc:2781] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:55.566784 29768 tablet_bootstrap.cc:492] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313: No bootstrap required, opened a new log
I20260806 20:13:55.567296 29768 ts_tablet_manager.cc:1397] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313: Time spent bootstrapping tablet: real 0.103s user 0.019s sys 0.008s
I20260806 20:13:55.568554 29769 tablet_bootstrap.cc:492] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8: No bootstrap required, opened a new log
I20260806 20:13:55.569037 29769 ts_tablet_manager.cc:1397] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.012s
I20260806 20:13:55.570550 29768 raft_consensus.cc:348] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.571312 29768 raft_consensus.cc:374] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.571590 29768 raft_consensus.cc:729] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.571621 29769 raft_consensus.cc:348] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.572446 29769 raft_consensus.cc:374] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.572772 29769 raft_consensus.cc:729] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.573333 29777 raft_consensus.cc:686] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:13:55.573503 29769 consensus_queue.cc:260] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.574230 29777 consensus_queue.cc:237] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.575989 29769 ts_tablet_manager.cc:1428] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:13:55.576733 29769 tablet_bootstrap.cc:492] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8: Bootstrap starting.
I20260806 20:13:55.572353 29768 consensus_queue.cc:260] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.578943 29768 ts_tablet_manager.cc:1428] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260806 20:13:55.579896 29768 tablet_bootstrap.cc:492] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313: Bootstrap starting.
I20260806 20:13:55.585978 29768 tablet_bootstrap.cc:654] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.594616 29768 tablet_bootstrap.cc:492] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313: No bootstrap required, opened a new log
I20260806 20:13:55.595201 29768 ts_tablet_manager.cc:1397] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.001s
I20260806 20:13:55.595728 29769 tablet_bootstrap.cc:654] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.598129 29768 raft_consensus.cc:348] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.598925 29768 raft_consensus.cc:374] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.599233 29768 raft_consensus.cc:729] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.599963 29768 consensus_queue.cc:260] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.602481 29768 ts_tablet_manager.cc:1428] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:13:55.604703 29452 catalog_manager.cc:5495] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: HEALTHY } } }
I20260806 20:13:55.610177 29769 tablet_bootstrap.cc:492] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8: No bootstrap required, opened a new log
I20260806 20:13:55.610646 29769 ts_tablet_manager.cc:1397] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.009s
I20260806 20:13:55.616259 29768 tablet_bootstrap.cc:492] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313: Bootstrap starting.
I20260806 20:13:55.618376 29769 raft_consensus.cc:348] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.619103 29769 raft_consensus.cc:374] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.619361 29769 raft_consensus.cc:729] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.619952 29769 consensus_queue.cc:260] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.622084 29769 ts_tablet_manager.cc:1428] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20260806 20:13:55.622887 29769 tablet_bootstrap.cc:492] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8: Bootstrap starting.
I20260806 20:13:55.630967 29768 tablet_bootstrap.cc:654] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.631599 29769 tablet_bootstrap.cc:654] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8: Neither blocks nor log segments found. Creating new log.
W20260806 20:13:55.637741 29502 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:13:55.640561 29769 tablet_bootstrap.cc:492] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8: No bootstrap required, opened a new log
I20260806 20:13:55.640936 29769 ts_tablet_manager.cc:1397] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260806 20:13:55.641742 29768 tablet_bootstrap.cc:492] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313: No bootstrap required, opened a new log
I20260806 20:13:55.642227 29768 ts_tablet_manager.cc:1397] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.003s
I20260806 20:13:55.642817 29769 raft_consensus.cc:348] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.643287 29769 raft_consensus.cc:374] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.643482 29769 raft_consensus.cc:729] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.643991 29769 consensus_queue.cc:260] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.644685 29768 raft_consensus.cc:348] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.645349 29768 raft_consensus.cc:374] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.645627 29768 raft_consensus.cc:729] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.646629 29769 ts_tablet_manager.cc:1428] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260806 20:13:55.646230 29768 consensus_queue.cc:260] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.650139 29777 raft_consensus.cc:482] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.650511 29768 ts_tablet_manager.cc:1428] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:13:55.650593 29777 raft_consensus.cc:504] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.651391 29768 tablet_bootstrap.cc:492] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313: Bootstrap starting.
I20260806 20:13:55.652554 29777 leader_election.cc:288] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:55.653622 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" is_pre_election: true
I20260806 20:13:55.654268 29641 raft_consensus.cc:2454] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 0.
I20260806 20:13:55.654358 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313" is_pre_election: true
I20260806 20:13:55.655385 29560 raft_consensus.cc:2454] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 0.
I20260806 20:13:55.656549 29682 leader_election.cc:302] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:13:55.657531 29777 raft_consensus.cc:2781] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:55.657651 29768 tablet_bootstrap.cc:654] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313: Neither blocks nor log segments found. Creating new log.
I20260806 20:13:55.658042 29777 raft_consensus.cc:482] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:55.658483 29777 raft_consensus.cc:3037] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.664947 29777 raft_consensus.cc:504] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.667867 29777 leader_election.cc:288] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:55.669191 29768 tablet_bootstrap.cc:492] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313: No bootstrap required, opened a new log
I20260806 20:13:55.669679 29768 ts_tablet_manager.cc:1397] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.003s
I20260806 20:13:55.670440 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:55.670830 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:55.671041 29641 raft_consensus.cc:3037] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.671368 29560 raft_consensus.cc:3037] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.672281 29768 raft_consensus.cc:348] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.672955 29768 raft_consensus.cc:374] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:13:55.673204 29768 raft_consensus.cc:729] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Initialized, Role: FOLLOWER
I20260806 20:13:55.673826 29768 consensus_queue.cc:260] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.677680 29641 raft_consensus.cc:2454] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 1.
I20260806 20:13:55.678112 29768 ts_tablet_manager.cc:1428] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:13:55.678889 29684 leader_election.cc:302] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:55.679842 29777 raft_consensus.cc:2781] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:55.680317 29777 raft_consensus.cc:686] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:13:55.681082 29777 consensus_queue.cc:237] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.688848 29560 raft_consensus.cc:2454] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 1.
I20260806 20:13:55.695995 29449 catalog_manager.cc:5495] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: HEALTHY } } }
I20260806 20:13:55.773626 29777 raft_consensus.cc:482] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.774072 29777 raft_consensus.cc:504] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.775689 29777 leader_election.cc:288] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:55.776532 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313" is_pre_election: true
I20260806 20:13:55.776880 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" is_pre_election: true
I20260806 20:13:55.777101 29560 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 0.
I20260806 20:13:55.777570 29641 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 0.
I20260806 20:13:55.778045 29682 leader_election.cc:302] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:13:55.778805 29777 raft_consensus.cc:2781] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:55.779079 29777 raft_consensus.cc:482] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:55.779338 29777 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.783560 29777 raft_consensus.cc:504] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.785290 29777 leader_election.cc:288] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:55.786222 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:55.786804 29560 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.787144 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:55.787680 29641 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.787536 29815 consensus_queue.cc:1035] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:55.793045 29560 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 1.
I20260806 20:13:55.793780 29641 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 1.
I20260806 20:13:55.794116 29682 leader_election.cc:302] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:13:55.794912 29821 raft_consensus.cc:2781] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:55.795400 29821 raft_consensus.cc:686] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:13:55.796232 29821 consensus_queue.cc:237] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.797693 29560 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.808740 29793 consensus_queue.cc:1035] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:13:55.811580 29796 consensus_queue.cc:1035] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:55.825847 29450 catalog_manager.cc:5495] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: HEALTHY } } }
I20260806 20:13:55.859380 29793 consensus_queue.cc:1035] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:55.861145 29839 raft_consensus.cc:482] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:55.861634 29839 raft_consensus.cc:504] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.868880 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133841d936c42e5a45fe986450328e3" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313" is_pre_election: true
I20260806 20:13:55.869684 29560 raft_consensus.cc:2454] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 0.
I20260806 20:13:55.870580 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133841d936c42e5a45fe986450328e3" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" is_pre_election: true
I20260806 20:13:55.871137 29601 leader_election.cc:302] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:55.872097 29793 raft_consensus.cc:2781] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:55.871253 29720 raft_consensus.cc:2454] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 0.
I20260806 20:13:55.872468 29793 raft_consensus.cc:482] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:55.872828 29793 raft_consensus.cc:3037] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.870746 29839 leader_election.cc:288] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.880515 29793 raft_consensus.cc:504] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.882155 29793 leader_election.cc:288] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:55.882877 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133841d936c42e5a45fe986450328e3" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:55.883400 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133841d936c42e5a45fe986450328e3" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:55.883402 29560 raft_consensus.cc:3037] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.883986 29720 raft_consensus.cc:3037] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.890092 29560 raft_consensus.cc:2454] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 1.
I20260806 20:13:55.891574 29796 consensus_queue.cc:1035] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:55.891655 29601 leader_election.cc:302] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:55.892745 29793 raft_consensus.cc:2781] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:55.892999 29721 raft_consensus.cc:3037] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:55.893294 29793 raft_consensus.cc:686] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:13:55.894259 29793 consensus_queue.cc:237] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:55.890941 29720 raft_consensus.cc:2454] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 1.
I20260806 20:13:55.903586 29796 consensus_queue.cc:1035] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:13:55.920362 29449 catalog_manager.cc:5495] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:55.983987 29796 consensus_queue.cc:1035] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:55.986212 29796 consensus_queue.cc:1035] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:56.027647 29821 raft_consensus.cc:482] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:13:56.028102 29821 raft_consensus.cc:504] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.029644 29821 leader_election.cc:288] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:56.030622 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3474e05293044f6cbbf938d0f54c9dd1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313" is_pre_election: true
I20260806 20:13:56.030773 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3474e05293044f6cbbf938d0f54c9dd1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" is_pre_election: true
I20260806 20:13:56.031148 29560 raft_consensus.cc:2454] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 0.
I20260806 20:13:56.031461 29641 raft_consensus.cc:2454] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 0.
I20260806 20:13:56.032356 29682 leader_election.cc:302] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:13:56.033036 29821 raft_consensus.cc:2781] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:13:56.033401 29821 raft_consensus.cc:482] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:13:56.033775 29821 raft_consensus.cc:3037] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:56.040588 29821 raft_consensus.cc:504] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.042647 29821 leader_election.cc:288] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:56.043489 29560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3474e05293044f6cbbf938d0f54c9dd1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:56.043722 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3474e05293044f6cbbf938d0f54c9dd1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:56.043994 29560 raft_consensus.cc:3037] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:56.044348 29641 raft_consensus.cc:3037] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:13:56.048555 29560 raft_consensus.cc:2454] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 1.
I20260806 20:13:56.049523 29682 leader_election.cc:302] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:13:56.050160 29821 raft_consensus.cc:2781] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:13:56.050546 29821 raft_consensus.cc:686] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Becoming Leader. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:13:56.050655 29641 raft_consensus.cc:2454] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 1.
I20260806 20:13:56.051365 29821 consensus_queue.cc:237] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.059755 29449 catalog_manager.cc:5495] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: HEALTHY } } }
I20260806 20:13:56.088877 29821 consensus_queue.cc:1035] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:56.098951 29849 consensus_queue.cc:1035] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:13:56.170751 29849 consensus_queue.cc:1035] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:56.183914 29849 consensus_queue.cc:1035] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:56.376852 29849 consensus_queue.cc:1035] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:56.391441 29849 consensus_queue.cc:1035] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:56.417843 29641 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
mode: GRACEFUL
new_leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:50378
I20260806 20:13:56.418399 29641 raft_consensus.cc:595] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Received request to transfer leadership to TS a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:13:56.433179 29720 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d"
dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
mode: GRACEFUL
new_leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:34820
I20260806 20:13:56.433771 29720 raft_consensus.cc:595] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Received request to transfer leadership to TS a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:13:56.438059 29720 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1"
dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
mode: GRACEFUL
new_leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:34820
I20260806 20:13:56.438540 29720 raft_consensus.cc:595] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Received request to transfer leadership to TS d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:13:56.441064 29793 consensus_queue.cc:1035] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:56.457756 29861 consensus_queue.cc:1035] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:13:56.464859 29793 raft_consensus.cc:979] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7: : Instructing follower a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:13:56.465360 29793 raft_consensus.cc:1067] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Signalling peer a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:13:56.468544 29560 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:40674
I20260806 20:13:56.470755 29560 raft_consensus.cc:482] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:13:56.471421 29560 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.472105 29856 consensus_queue.cc:1035] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:13:56.478485 29560 raft_consensus.cc:504] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.481527 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:56.482136 29641 raft_consensus.cc:3032] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:13:56.482492 29641 raft_consensus.cc:729] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:13:56.483490 29641 consensus_queue.cc:260] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.484896 29641 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.485872 29558 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
mode: GRACEFUL
new_leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:39838
I20260806 20:13:56.486627 29560 leader_election.cc:288] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 2 election: Requested vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:56.488629 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:56.489171 29721 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.491298 29558 raft_consensus.cc:595] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Received request to transfer leadership to TS d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:13:56.492216 29641 raft_consensus.cc:2454] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 2.
I20260806 20:13:56.493582 29522 leader_election.cc:302] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:56.494460 29795 raft_consensus.cc:2781] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:13:56.494939 29795 raft_consensus.cc:686] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Becoming Leader. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:13:56.495886 29795 consensus_queue.cc:237] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.498838 29720 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc"
dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
mode: GRACEFUL
new_leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:34820
I20260806 20:13:56.499418 29720 raft_consensus.cc:595] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Received request to transfer leadership to TS d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:13:56.500989 29721 raft_consensus.cc:2454] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 2.
I20260806 20:13:56.505873 29856 consensus_queue.cc:1035] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:13:56.506569 29449 catalog_manager.cc:5495] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 reported cstate change: term changed from 1 to 2, leader changed from d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194) to a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193). New cstate: current_term: 2 leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:56.516352 29849 raft_consensus.cc:979] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8: : Instructing follower a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:13:56.516904 29777 raft_consensus.cc:1067] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Signalling peer a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:13:56.518947 29559 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:40680
I20260806 20:13:56.519482 29559 raft_consensus.cc:482] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:13:56.519804 29559 raft_consensus.cc:3037] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.526592 29559 raft_consensus.cc:504] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.528721 29559 leader_election.cc:288] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 2 election: Requested vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:56.529884 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:56.530409 29721 raft_consensus.cc:3032] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:13:56.530637 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:56.530652 29721 raft_consensus.cc:729] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:13:56.531204 29641 raft_consensus.cc:3037] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.531605 29721 consensus_queue.cc:260] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.532891 29721 raft_consensus.cc:3037] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.534803 29796 raft_consensus.cc:979] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313: : Instructing follower d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:13:56.535223 29795 raft_consensus.cc:1067] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Signalling peer d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:13:56.536861 29641 raft_consensus.cc:2454] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 2.
I20260806 20:13:56.536938 29640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:49498
I20260806 20:13:56.537568 29640 raft_consensus.cc:482] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:13:56.537940 29640 raft_consensus.cc:3037] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.538022 29522 leader_election.cc:302] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:56.538815 29795 raft_consensus.cc:2781] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:13:56.539181 29795 raft_consensus.cc:686] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Becoming Leader. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:13:56.539450 29721 raft_consensus.cc:2454] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 2.
I20260806 20:13:56.540069 29795 consensus_queue.cc:237] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.544322 29640 raft_consensus.cc:504] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.547056 29640 leader_election.cc:288] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 2 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:56.547833 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:56.548151 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:56.547839 29449 catalog_manager.cc:5495] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 reported cstate change: term changed from 1 to 2, leader changed from 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195) to a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193). New cstate: current_term: 2 leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:56.548678 29721 raft_consensus.cc:3037] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.548877 29559 raft_consensus.cc:3032] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:13:56.549224 29559 raft_consensus.cc:729] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:13:56.550086 29559 consensus_queue.cc:260] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.551337 29559 raft_consensus.cc:3037] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.555182 29721 raft_consensus.cc:2454] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 2.
I20260806 20:13:56.556366 29604 leader_election.cc:302] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:56.557240 29861 raft_consensus.cc:2781] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:13:56.557746 29861 raft_consensus.cc:686] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:13:56.558373 29559 raft_consensus.cc:2454] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 2.
I20260806 20:13:56.558669 29861 consensus_queue.cc:237] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.565423 29449 catalog_manager.cc:5495] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 1 to 2, leader changed from a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193) to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:56.772399 29849 raft_consensus.cc:979] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8: : Instructing follower d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:13:56.772882 29856 raft_consensus.cc:1067] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Signalling peer d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:13:56.774444 29640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:49506
I20260806 20:13:56.774940 29640 raft_consensus.cc:482] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:13:56.775229 29640 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.780194 29640 raft_consensus.cc:504] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.781915 29640 leader_election.cc:288] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 2 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:56.782882 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:56.783061 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:56.783469 29559 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.783576 29721 raft_consensus.cc:3032] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:13:56.783901 29721 raft_consensus.cc:729] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:13:56.784498 29721 consensus_queue.cc:260] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.785403 29721 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.788403 29559 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 2.
I20260806 20:13:56.789556 29601 leader_election.cc:302] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:56.790086 29721 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 2.
I20260806 20:13:56.790300 29861 raft_consensus.cc:2781] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:13:56.790807 29861 raft_consensus.cc:686] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:13:56.791540 29861 consensus_queue.cc:237] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.797729 29449 catalog_manager.cc:5495] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 1 to 2, leader changed from 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195) to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:56.802850 29849 raft_consensus.cc:979] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8: : Instructing follower d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:13:56.803251 29856 raft_consensus.cc:1067] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Signalling peer d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:13:56.805058 29640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:49506
I20260806 20:13:56.805612 29640 raft_consensus.cc:482] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:13:56.805950 29640 raft_consensus.cc:3037] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.811094 29640 raft_consensus.cc:504] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.813086 29640 leader_election.cc:288] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 2 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:13:56.813953 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:56.814100 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:13:56.814437 29559 raft_consensus.cc:3037] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.814574 29721 raft_consensus.cc:3032] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:13:56.814896 29721 raft_consensus.cc:729] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:13:56.815546 29721 consensus_queue.cc:260] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.816433 29721 raft_consensus.cc:3037] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:56.819377 29559 raft_consensus.cc:2454] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 2.
I20260806 20:13:56.820432 29601 leader_election.cc:302] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:56.820811 29721 raft_consensus.cc:2454] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 2.
I20260806 20:13:56.821277 29861 raft_consensus.cc:2781] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:13:56.821787 29861 raft_consensus.cc:686] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:13:56.822506 29861 consensus_queue.cc:237] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:56.828176 29449 catalog_manager.cc:5495] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 1 to 2, leader changed from 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195) to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:13:56.881134 29640 raft_consensus.cc:1261] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:56.882400 29795 consensus_queue.cc:1035] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:56.890303 29721 raft_consensus.cc:1261] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:56.892275 29795 consensus_queue.cc:1035] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:57.058080 29721 raft_consensus.cc:1261] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:57.059245 29862 consensus_queue.cc:1035] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:57.067260 29559 raft_consensus.cc:1261] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:57.068828 29793 consensus_queue.cc:1035] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:57.086189 29641 raft_consensus.cc:1261] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:57.087235 29815 consensus_queue.cc:1035] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:57.093058 29721 raft_consensus.cc:1261] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:57.094354 29795 consensus_queue.cc:1035] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:57.218670 29559 raft_consensus.cc:1261] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:57.219945 29793 consensus_queue.cc:1035] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:57.223197 29720 raft_consensus.cc:1261] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:57.224704 29793 consensus_queue.cc:1035] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:57.229415 29558 raft_consensus.cc:1261] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:57.231086 29861 consensus_queue.cc:1035] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:57.244531 29720 raft_consensus.cc:1261] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:13:57.245839 29793 consensus_queue.cc:1035] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:13:59.536121 29419 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260806 20:13:59.537022 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:13:59.558784 29641 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
mode: GRACEFUL
new_leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
from {username='slave'} at 127.0.0.1:50384
I20260806 20:13:59.559333 29641 raft_consensus.cc:595] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Received request to transfer leadership to TS 96cbcc4fe5f841d798d8b6131fafd3f8
I20260806 20:13:59.571799 29559 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
mode: GRACEFUL
new_leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
from {username='slave'} at 127.0.0.1:39844
I20260806 20:13:59.572325 29559 raft_consensus.cc:595] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Received request to transfer leadership to TS 96cbcc4fe5f841d798d8b6131fafd3f8
I20260806 20:13:59.573361 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 20:13:59.573848 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:13:59.766624 29878 raft_consensus.cc:979] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7: : Instructing follower 96cbcc4fe5f841d798d8b6131fafd3f8 to start an election
I20260806 20:13:59.767040 29873 raft_consensus.cc:1067] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Signalling peer 96cbcc4fe5f841d798d8b6131fafd3f8 to start an election
I20260806 20:13:59.768445 29720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1"
dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
from {username='slave'} at 127.0.0.1:35104
I20260806 20:13:59.768923 29720 raft_consensus.cc:482] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:13:59.769207 29720 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:13:59.773569 29720 raft_consensus.cc:504] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:59.775130 29720 leader_election.cc:288] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 3 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:59.776124 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:59.776060 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:59.776782 29641 raft_consensus.cc:3032] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:13:59.776859 29559 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:13:59.777073 29641 raft_consensus.cc:729] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:13:59.777748 29641 consensus_queue.cc:260] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:59.778656 29641 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:13:59.782303 29559 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 3.
I20260806 20:13:59.782845 29641 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 3.
I20260806 20:13:59.783407 29682 leader_election.cc:302] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:13:59.784097 29856 raft_consensus.cc:2781] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:13:59.785118 29856 raft_consensus.cc:686] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 LEADER]: Becoming Leader. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:13:59.785784 29856 consensus_queue.cc:237] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:13:59.794787 29452 catalog_manager.cc:5495] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 reported cstate change: term changed from 2 to 3, leader changed from d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194) to 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: HEALTHY } } }
I20260806 20:13:59.960050 29885 raft_consensus.cc:979] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313: : Instructing follower 96cbcc4fe5f841d798d8b6131fafd3f8 to start an election
I20260806 20:13:59.960451 29795 raft_consensus.cc:1067] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Signalling peer 96cbcc4fe5f841d798d8b6131fafd3f8 to start an election
I20260806 20:13:59.961894 29720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f"
dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
from {username='slave'} at 127.0.0.1:35116
I20260806 20:13:59.962399 29720 raft_consensus.cc:482] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:13:59.962680 29720 raft_consensus.cc:3037] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:59.968124 29720 raft_consensus.cc:504] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:59.970314 29720 leader_election.cc:288] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 2 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:13:59.970585 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:13:59.970815 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:13:59.971382 29559 raft_consensus.cc:3032] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:13:59.971426 29641 raft_consensus.cc:3037] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:59.971805 29559 raft_consensus.cc:729] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:13:59.972582 29559 consensus_queue.cc:260] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:59.973692 29559 raft_consensus.cc:3037] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:13:59.976046 29641 raft_consensus.cc:2454] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 2.
I20260806 20:13:59.976974 29684 leader_election.cc:302] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:13:59.977658 29856 raft_consensus.cc:2781] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:13:59.978019 29856 raft_consensus.cc:686] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 LEADER]: Becoming Leader. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:13:59.978624 29559 raft_consensus.cc:2454] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 2.
I20260806 20:13:59.978744 29856 consensus_queue.cc:237] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:13:59.985371 29452 catalog_manager.cc:5495] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 reported cstate change: term changed from 1 to 2, leader changed from a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193) to 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195). New cstate: current_term: 2 leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:00.317360 29559 raft_consensus.cc:1261] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Refusing update from remote peer 96cbcc4fe5f841d798d8b6131fafd3f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:00.318661 29856 consensus_queue.cc:1035] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:00.328984 29641 raft_consensus.cc:1261] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Refusing update from remote peer 96cbcc4fe5f841d798d8b6131fafd3f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:00.332314 29879 consensus_queue.cc:1035] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:00.362600 29559 raft_consensus.cc:1261] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Refusing update from remote peer 96cbcc4fe5f841d798d8b6131fafd3f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:00.364041 29879 consensus_queue.cc:1035] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:00.378132 29641 raft_consensus.cc:1261] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Refusing update from remote peer 96cbcc4fe5f841d798d8b6131fafd3f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:00.379371 29856 consensus_queue.cc:1035] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:00.574762 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:00.579596 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:00.579948 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:01.580796 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:01.585827 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:01.586212 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:02.587330 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:02.595850 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:02.596356 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:03.597218 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:03.601810 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:03.602159 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:04.602937 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:04.607573 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:04.607935 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:05.608665 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:05.613613 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:05.614037 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:06.614883 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:06.619752 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:06.620141 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:07.620954 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:07.626235 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:07.626648 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:08.627351 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:08.631861 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:08.632200 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:09.658067 29419 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260806 20:14:09.662470 29721 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f"
dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
mode: GRACEFUL
new_leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:34820
I20260806 20:14:09.663216 29721 raft_consensus.cc:595] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 LEADER]: Received request to transfer leadership to TS d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:14:09.668300 29559 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
mode: GRACEFUL
new_leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:39838
I20260806 20:14:09.668784 29559 raft_consensus.cc:595] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Received request to transfer leadership to TS d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:14:09.673265 29559 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
mode: GRACEFUL
new_leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:39838
I20260806 20:14:09.673862 29559 raft_consensus.cc:595] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Received request to transfer leadership to TS d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:14:09.674220 29903 raft_consensus.cc:979] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8: : Instructing follower d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.674757 29879 raft_consensus.cc:1067] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 LEADER]: Signalling peer d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.676158 29640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:49506
I20260806 20:14:09.676577 29640 raft_consensus.cc:482] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:09.676985 29640 raft_consensus.cc:3037] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:09.678408 29559 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
mode: GRACEFUL
new_leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:39838
I20260806 20:14:09.678833 29559 raft_consensus.cc:595] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Received request to transfer leadership to TS d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:14:09.683208 29721 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1"
dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
mode: GRACEFUL
new_leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:34820
I20260806 20:14:09.683821 29721 raft_consensus.cc:595] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 LEADER]: Received request to transfer leadership to TS d5ad1e1b5e874ccea53daf19b90f22a7
I20260806 20:14:09.683252 29640 raft_consensus.cc:504] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:14:09.685365 29640 leader_election.cc:288] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 3 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:14:09.686477 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:14:09.687142 29559 raft_consensus.cc:3037] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:09.686908 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d150978500404c0c84d2c1be4767b06f" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:14:09.687608 29721 raft_consensus.cc:3032] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:09.687932 29721 raft_consensus.cc:729] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:14:09.688638 29721 consensus_queue.cc:260] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:14:09.689935 29721 raft_consensus.cc:3037] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:09.692709 29559 raft_consensus.cc:2454] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 3.
I20260806 20:14:09.693790 29601 leader_election.cc:302] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:14:09.694636 29873 raft_consensus.cc:2781] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:09.694958 29873 raft_consensus.cc:686] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:09.695611 29873 consensus_queue.cc:237] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } }
I20260806 20:14:09.696743 29721 raft_consensus.cc:2454] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 3.
I20260806 20:14:09.702349 29449 catalog_manager.cc:5495] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 2 to 3, leader changed from 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195) to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 3 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:09.735988 29909 raft_consensus.cc:979] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313: : Instructing follower d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.736457 29885 raft_consensus.cc:1067] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Signalling peer d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.737820 29640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:49498
I20260806 20:14:09.738281 29640 raft_consensus.cc:482] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:09.738550 29640 raft_consensus.cc:3037] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:09.743047 29640 raft_consensus.cc:504] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.744601 29640 leader_election.cc:288] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 3 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:14:09.745505 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:14:09.745659 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ff2293355b476b85beb264fc24f12d" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:14:09.746083 29559 raft_consensus.cc:3032] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:09.746145 29721 raft_consensus.cc:3037] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:09.746373 29559 raft_consensus.cc:729] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:14:09.747006 29559 consensus_queue.cc:260] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.747970 29559 raft_consensus.cc:3037] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:09.751292 29721 raft_consensus.cc:2454] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 3.
I20260806 20:14:09.752214 29559 raft_consensus.cc:2454] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 3.
I20260806 20:14:09.752490 29604 leader_election.cc:302] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:14:09.753186 29873 raft_consensus.cc:2781] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:09.753605 29873 raft_consensus.cc:686] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:09.754187 29873 consensus_queue.cc:237] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.760598 29449 catalog_manager.cc:5495] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 2 to 3, leader changed from a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193) to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 3 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:09.849118 29909 raft_consensus.cc:979] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313: : Instructing follower d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.849575 29885 raft_consensus.cc:1067] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Signalling peer d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.851001 29640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:49498
I20260806 20:14:09.851452 29640 raft_consensus.cc:482] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:09.851730 29640 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:09.856256 29640 raft_consensus.cc:504] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.857800 29640 leader_election.cc:288] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 3 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:14:09.858605 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:14:09.858790 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:14:09.859189 29559 raft_consensus.cc:3032] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:09.859272 29721 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:09.859517 29559 raft_consensus.cc:729] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:14:09.860260 29559 consensus_queue.cc:260] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.861481 29559 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:09.865760 29721 raft_consensus.cc:2454] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 3.
I20260806 20:14:09.866787 29604 leader_election.cc:302] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:14:09.867213 29559 raft_consensus.cc:2454] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 3.
I20260806 20:14:09.867431 29873 raft_consensus.cc:2781] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:09.867870 29873 raft_consensus.cc:686] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:09.868546 29873 consensus_queue.cc:237] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.874955 29449 catalog_manager.cc:5495] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 2 to 3, leader changed from a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193) to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 3 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:09.932729 29909 raft_consensus.cc:979] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313: : Instructing follower d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.933157 29909 raft_consensus.cc:1067] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Signalling peer d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.934711 29640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:49498
I20260806 20:14:09.935247 29640 raft_consensus.cc:482] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:09.935621 29640 raft_consensus.cc:3037] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:09.936674 29903 raft_consensus.cc:979] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8: : Instructing follower d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.937167 29879 raft_consensus.cc:1067] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 LEADER]: Signalling peer d5ad1e1b5e874ccea53daf19b90f22a7 to start an election
I20260806 20:14:09.938807 29641 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
from {username='slave'} at 127.0.0.1:49506
I20260806 20:14:09.939291 29641 raft_consensus.cc:482] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:09.939587 29641 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:09.941895 29640 raft_consensus.cc:504] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.943570 29640 leader_election.cc:288] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 2 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:14:09.944265 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:14:09.944857 29559 raft_consensus.cc:3032] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:09.945088 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:14:09.945205 29559 raft_consensus.cc:729] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:14:09.945202 29641 raft_consensus.cc:504] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.945660 29721 raft_consensus.cc:3037] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:09.946790 29641 leader_election.cc:288] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 4 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:14:09.946095 29559 consensus_queue.cc:260] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.947749 29558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:14:09.948040 29559 raft_consensus.cc:3037] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:09.948827 29558 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:09.952199 29721 raft_consensus.cc:2454] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 2.
I20260806 20:14:09.953608 29604 leader_election.cc:302] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:14:09.954010 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a8b884e76a547bf8ff5a7c371f629a1" candidate_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:14:09.954452 29558 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 4.
I20260806 20:14:09.954680 29721 raft_consensus.cc:3032] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:14:09.954536 29559 raft_consensus.cc:2454] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 2.
I20260806 20:14:09.954484 29873 raft_consensus.cc:2781] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:09.955159 29721 raft_consensus.cc:729] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:14:09.955724 29601 leader_election.cc:302] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a10e98dae4fb4f52b56f05afd3f86313, d5ad1e1b5e874ccea53daf19b90f22a7; no voters:
I20260806 20:14:09.956082 29873 raft_consensus.cc:686] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:09.955935 29721 consensus_queue.cc:260] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.956558 29902 raft_consensus.cc:2781] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:14:09.956784 29873 consensus_queue.cc:237] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.957077 29721 raft_consensus.cc:3037] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:09.957954 29902 raft_consensus.cc:686] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 4 LEADER]: Becoming Leader. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:09.958772 29902 consensus_queue.cc:237] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:09.963436 29721 raft_consensus.cc:2454] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ad1e1b5e874ccea53daf19b90f22a7 in term 4.
I20260806 20:14:09.964922 29449 catalog_manager.cc:5495] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 1 to 2, leader changed from a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193) to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:09.976524 29449 catalog_manager.cc:5495] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 reported cstate change: term changed from 3 to 4, leader changed from 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195) to d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194). New cstate: current_term: 4 leader_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:10.167891 29559 raft_consensus.cc:1261] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:10.168982 29873 consensus_queue.cc:1035] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:10.178128 29721 raft_consensus.cc:1261] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:10.180002 29873 consensus_queue.cc:1035] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:10.333658 29559 raft_consensus.cc:1261] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:10.334725 29873 consensus_queue.cc:1035] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:10.345547 29720 raft_consensus.cc:1261] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:10.348126 29912 consensus_queue.cc:1035] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:10.348920 29721 raft_consensus.cc:1261] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:10.350471 29912 consensus_queue.cc:1035] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:10.359360 29559 raft_consensus.cc:1261] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:10.360782 29873 consensus_queue.cc:1035] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:10.401839 29559 raft_consensus.cc:1261] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:10.402892 29873 consensus_queue.cc:1035] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:10.410745 29721 raft_consensus.cc:1261] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:10.412194 29902 consensus_queue.cc:1035] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:10.491112 29559 raft_consensus.cc:1261] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 4 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:14:10.492223 29902 consensus_queue.cc:1035] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:14:10.500267 29721 raft_consensus.cc:1261] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 4 FOLLOWER]: Refusing update from remote peer d5ad1e1b5e874ccea53daf19b90f22a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:14:10.501753 29873 consensus_queue.cc:1035] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:14:12.724610 29419 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260806 20:14:12.725332 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:12.732218 29641 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
mode: GRACEFUL
new_leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:50384
I20260806 20:14:12.732903 29641 raft_consensus.cc:595] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Received request to transfer leadership to TS a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:14:12.735258 29641 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
mode: GRACEFUL
new_leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
from {username='slave'} at 127.0.0.1:50384
I20260806 20:14:12.735694 29641 raft_consensus.cc:595] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Received request to transfer leadership to TS 96cbcc4fe5f841d798d8b6131fafd3f8
I20260806 20:14:12.737656 29641 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
mode: GRACEFUL
new_leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:50384
I20260806 20:14:12.738039 29641 raft_consensus.cc:595] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Received request to transfer leadership to TS a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:14:12.739843 29641 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d133841d936c42e5a45fe986450328e3"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
mode: GRACEFUL
new_leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:50384
I20260806 20:14:12.740226 29641 raft_consensus.cc:595] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Received request to transfer leadership to TS a10e98dae4fb4f52b56f05afd3f86313
I20260806 20:14:12.742071 29641 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e"
dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
mode: GRACEFUL
new_leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
from {username='slave'} at 127.0.0.1:50384
I20260806 20:14:12.742456 29641 raft_consensus.cc:595] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 LEADER]: Received request to transfer leadership to TS 96cbcc4fe5f841d798d8b6131fafd3f8
I20260806 20:14:12.743250 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:14:12.743652 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:12.781301 29902 raft_consensus.cc:979] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7: : Instructing follower a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:14:12.781848 29933 raft_consensus.cc:1067] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Signalling peer a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:14:12.783353 29558 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:40674
I20260806 20:14:12.783818 29558 raft_consensus.cc:482] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:12.784134 29558 raft_consensus.cc:3037] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:12.791832 29558 raft_consensus.cc:504] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:12.793304 29902 raft_consensus.cc:979] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7: : Instructing follower 96cbcc4fe5f841d798d8b6131fafd3f8 to start an election
I20260806 20:14:12.793853 29933 raft_consensus.cc:1067] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 LEADER]: Signalling peer 96cbcc4fe5f841d798d8b6131fafd3f8 to start an election
I20260806 20:14:12.794924 29558 leader_election.cc:288] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 3 election: Requested vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:14:12.796274 29721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:14:12.796818 29721 raft_consensus.cc:3037] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:12.801257 29721 raft_consensus.cc:2454] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 3.
I20260806 20:14:12.802537 29524 leader_election.cc:302] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:14:12.802665 29720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e"
dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
from {username='slave'} at 127.0.0.1:35104
I20260806 20:14:12.802559 29641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14689f97dadc4b0e95a25cedc9f172cc" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:14:12.803287 29720 raft_consensus.cc:482] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:12.803336 29641 raft_consensus.cc:3032] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:12.803750 29720 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:12.803809 29641 raft_consensus.cc:729] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:12.804388 29934 raft_consensus.cc:2781] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:12.804756 29641 consensus_queue.cc:260] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:12.805486 29934 raft_consensus.cc:686] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 3 LEADER]: Becoming Leader. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:14:12.806851 29641 raft_consensus.cc:3037] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:12.806387 29934 consensus_queue.cc:237] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:12.809336 29720 raft_consensus.cc:504] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:12.812216 29720 leader_election.cc:288] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 4 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:14:12.812666 29558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:14:12.812980 29640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e60d5891ab409ea2a360dd221d3d1e" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:14:12.813304 29558 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:12.813556 29640 raft_consensus.cc:3032] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:14:12.813859 29640 raft_consensus.cc:729] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:12.814679 29640 consensus_queue.cc:260] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:12.814937 29452 catalog_manager.cc:5495] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 reported cstate change: term changed from 2 to 3, leader changed from d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194) to a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193). New cstate: current_term: 3 leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:12.816037 29640 raft_consensus.cc:3037] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:12.818130 29641 raft_consensus.cc:2454] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 3.
I20260806 20:14:12.820858 29558 raft_consensus.cc:2454] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 4.
I20260806 20:14:12.822374 29682 leader_election.cc:302] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:14:12.823102 29932 raft_consensus.cc:2781] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:14:12.823890 29640 raft_consensus.cc:2454] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 4.
I20260806 20:14:12.824335 29932 raft_consensus.cc:686] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 4 LEADER]: Becoming Leader. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:14:12.825341 29932 consensus_queue.cc:237] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:12.835238 29449 catalog_manager.cc:5495] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 reported cstate change: term changed from 3 to 4, leader changed from d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194) to 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195). New cstate: current_term: 4 leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:13.068168 29933 raft_consensus.cc:979] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7: : Instructing follower a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:14:13.068686 29944 raft_consensus.cc:1067] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Signalling peer a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:14:13.070411 29559 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:40674
I20260806 20:14:13.070847 29559 raft_consensus.cc:482] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:13.071094 29559 raft_consensus.cc:3037] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:13.075657 29559 raft_consensus.cc:504] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:13.077004 29559 leader_election.cc:288] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 3 election: Requested vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:14:13.078006 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:14:13.078018 29640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd4523c8fcd4e3c950fe41d0a908f19" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:14:13.078505 29720 raft_consensus.cc:3037] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:13.078568 29640 raft_consensus.cc:3032] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:13.078868 29640 raft_consensus.cc:729] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:13.079444 29640 consensus_queue.cc:260] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:13.080367 29640 raft_consensus.cc:3037] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:13.082886 29720 raft_consensus.cc:2454] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 3.
I20260806 20:14:13.083782 29524 leader_election.cc:302] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:14:13.084565 29934 raft_consensus.cc:2781] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:13.084930 29934 raft_consensus.cc:686] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 LEADER]: Becoming Leader. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:14:13.085106 29640 raft_consensus.cc:2454] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 3.
I20260806 20:14:13.085726 29934 consensus_queue.cc:237] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:13.092417 29449 catalog_manager.cc:5495] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 reported cstate change: term changed from 2 to 3, leader changed from d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194) to a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193). New cstate: current_term: 3 leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:13.099550 29933 raft_consensus.cc:979] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7: : Instructing follower a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:14:13.100026 29944 raft_consensus.cc:1067] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Signalling peer a10e98dae4fb4f52b56f05afd3f86313 to start an election
I20260806 20:14:13.101301 29559 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d133841d936c42e5a45fe986450328e3"
dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
from {username='slave'} at 127.0.0.1:40674
I20260806 20:14:13.101766 29559 raft_consensus.cc:482] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:13.102000 29559 raft_consensus.cc:3037] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:13.106076 29559 raft_consensus.cc:504] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:13.107429 29559 leader_election.cc:288] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 2 election: Requested vote from peers d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689), 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195:42515)
I20260806 20:14:13.108258 29640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133841d936c42e5a45fe986450328e3" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:14:13.108403 29720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133841d936c42e5a45fe986450328e3" candidate_uuid: "a10e98dae4fb4f52b56f05afd3f86313" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
I20260806 20:14:13.108757 29640 raft_consensus.cc:3032] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:13.108889 29720 raft_consensus.cc:3037] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:13.109040 29640 raft_consensus.cc:729] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:13.109758 29640 consensus_queue.cc:260] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:13.110731 29640 raft_consensus.cc:3037] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:13.113095 29720 raft_consensus.cc:2454] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 2.
I20260806 20:14:13.113934 29524 leader_election.cc:302] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:14:13.114651 29934 raft_consensus.cc:2781] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:13.115042 29640 raft_consensus.cc:2454] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a10e98dae4fb4f52b56f05afd3f86313 in term 2.
I20260806 20:14:13.115139 29934 raft_consensus.cc:686] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Becoming Leader. State: Replica: a10e98dae4fb4f52b56f05afd3f86313, State: Running, Role: LEADER
I20260806 20:14:13.115988 29934 consensus_queue.cc:237] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:13.122913 29449 catalog_manager.cc:5495] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 reported cstate change: term changed from 1 to 2, leader changed from d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194) to a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193). New cstate: current_term: 2 leader_uuid: "a10e98dae4fb4f52b56f05afd3f86313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:13.187775 29720 raft_consensus.cc:1261] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:13.189029 29934 consensus_queue.cc:1035] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:13.199980 29640 raft_consensus.cc:1261] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:13.201409 29951 consensus_queue.cc:1035] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:14:13.213310 29933 raft_consensus.cc:979] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7: : Instructing follower 96cbcc4fe5f841d798d8b6131fafd3f8 to start an election
I20260806 20:14:13.213807 29933 raft_consensus.cc:1067] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Signalling peer 96cbcc4fe5f841d798d8b6131fafd3f8 to start an election
I20260806 20:14:13.215114 29720 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63"
dest_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8"
from {username='slave'} at 127.0.0.1:35104
I20260806 20:14:13.215624 29720 raft_consensus.cc:482] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:13.215943 29720 raft_consensus.cc:3037] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:13.220501 29720 raft_consensus.cc:504] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:13.221931 29720 leader_election.cc:288] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 3 election: Requested vote from peers a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817), d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689)
I20260806 20:14:13.222589 29559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a10e98dae4fb4f52b56f05afd3f86313"
I20260806 20:14:13.222886 29640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b08a45d83e140d7b6a07558f4da9a63" candidate_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7"
I20260806 20:14:13.223142 29559 raft_consensus.cc:3037] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:13.223379 29640 raft_consensus.cc:3032] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:13.223649 29640 raft_consensus.cc:729] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d5ad1e1b5e874ccea53daf19b90f22a7, State: Running, Role: LEADER
I20260806 20:14:13.224346 29640 consensus_queue.cc:260] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:13.225615 29640 raft_consensus.cc:3037] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:13.229228 29559 raft_consensus.cc:2454] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 3.
I20260806 20:14:13.230317 29682 leader_election.cc:302] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cbcc4fe5f841d798d8b6131fafd3f8, a10e98dae4fb4f52b56f05afd3f86313; no voters:
I20260806 20:14:13.230968 29932 raft_consensus.cc:2781] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:13.232674 29932 raft_consensus.cc:686] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 LEADER]: Becoming Leader. State: Replica: 96cbcc4fe5f841d798d8b6131fafd3f8, State: Running, Role: LEADER
I20260806 20:14:13.232746 29640 raft_consensus.cc:2454] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cbcc4fe5f841d798d8b6131fafd3f8 in term 3.
I20260806 20:14:13.233577 29932 consensus_queue.cc:237] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } }
I20260806 20:14:13.241267 29452 catalog_manager.cc:5495] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 reported cstate change: term changed from 2 to 3, leader changed from d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194) to 96cbcc4fe5f841d798d8b6131fafd3f8 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:13.353847 29559 raft_consensus.cc:1261] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 4 FOLLOWER]: Refusing update from remote peer 96cbcc4fe5f841d798d8b6131fafd3f8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:14:13.355020 29932 consensus_queue.cc:1035] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:14:13.365278 29640 raft_consensus.cc:1261] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 4 FOLLOWER]: Refusing update from remote peer 96cbcc4fe5f841d798d8b6131fafd3f8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:14:13.367561 29956 consensus_queue.cc:1035] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:14:13.466184 29720 raft_consensus.cc:1261] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:13.467610 29951 consensus_queue.cc:1035] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:13.476980 29640 raft_consensus.cc:1261] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:13.478714 29951 consensus_queue.cc:1035] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:13.542997 29720 raft_consensus.cc:1261] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:13.544143 29951 consensus_queue.cc:1035] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cbcc4fe5f841d798d8b6131fafd3f8" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:13.552623 29640 raft_consensus.cc:1261] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Refusing update from remote peer a10e98dae4fb4f52b56f05afd3f86313: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:13.554191 29935 consensus_queue.cc:1035] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:14:13.656003 29640 raft_consensus.cc:1261] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Refusing update from remote peer 96cbcc4fe5f841d798d8b6131fafd3f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:13.657982 29938 consensus_queue.cc:1035] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5ad1e1b5e874ccea53daf19b90f22a7" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:13.666363 29559 raft_consensus.cc:1261] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Refusing update from remote peer 96cbcc4fe5f841d798d8b6131fafd3f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:13.667654 29938 consensus_queue.cc:1035] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10e98dae4fb4f52b56f05afd3f86313" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:13.744531 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:13.750367 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:13.750751 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:14.751534 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:14.756093 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:14.756487 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:15.757350 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:15.762491 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:15.762903 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:16.763798 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:16.768589 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:16.768961 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:17.769734 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:17.774737 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:17.775130 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:18.775808 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:18.780256 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:18.780587 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:19.781400 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:19.786278 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:19.786640 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:20.787417 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:20.792021 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:20.792387 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:21.793226 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:21.797930 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:21.798308 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:22.825009 29419 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260806 20:14:22.825691 29419 tablet_server.cc:178] TabletServer@127.28.186.193:0 shutting down...
I20260806 20:14:22.852871 29419 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:14:22.853758 29419 tablet_replica.cc:330] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313: stopping tablet replica
I20260806 20:14:22.854750 29419 raft_consensus.cc:2229] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:22.855556 29419 raft_consensus.cc:2258] T 3474e05293044f6cbbf938d0f54c9dd1 P a10e98dae4fb4f52b56f05afd3f86313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:22.857982 29419 tablet_replica.cc:330] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313: stopping tablet replica
I20260806 20:14:22.858435 29419 raft_consensus.cc:2229] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:22.858800 29419 raft_consensus.cc:2258] T b3ff2293355b476b85beb264fc24f12d P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:22.860378 29419 tablet_replica.cc:330] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313: stopping tablet replica
I20260806 20:14:22.860816 29419 raft_consensus.cc:2229] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:14:22.861658 29419 raft_consensus.cc:2258] T d133841d936c42e5a45fe986450328e3 P a10e98dae4fb4f52b56f05afd3f86313 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:22.863206 29419 tablet_replica.cc:330] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313: stopping tablet replica
I20260806 20:14:22.863631 29419 raft_consensus.cc:2229] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:22.864094 29419 raft_consensus.cc:2258] T f6e60d5891ab409ea2a360dd221d3d1e P a10e98dae4fb4f52b56f05afd3f86313 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:22.865800 29419 tablet_replica.cc:330] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313: stopping tablet replica
I20260806 20:14:22.866261 29419 raft_consensus.cc:2229] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:14:22.867012 29419 raft_consensus.cc:2258] T 14689f97dadc4b0e95a25cedc9f172cc P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:22.868551 29419 tablet_replica.cc:330] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313: stopping tablet replica
I20260806 20:14:22.868975 29419 raft_consensus.cc:2229] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:22.869400 29419 raft_consensus.cc:2258] T 3a8b884e76a547bf8ff5a7c371f629a1 P a10e98dae4fb4f52b56f05afd3f86313 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:22.870927 29419 tablet_replica.cc:330] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313: stopping tablet replica
I20260806 20:14:22.871371 29419 raft_consensus.cc:2229] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:14:22.872121 29419 raft_consensus.cc:2258] T 8bd4523c8fcd4e3c950fe41d0a908f19 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:22.873809 29419 tablet_replica.cc:330] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313: stopping tablet replica
I20260806 20:14:22.874259 29419 raft_consensus.cc:2229] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:22.874678 29419 raft_consensus.cc:2258] T 2b08a45d83e140d7b6a07558f4da9a63 P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:22.876117 29419 tablet_replica.cc:330] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313: stopping tablet replica
I20260806 20:14:22.876523 29419 raft_consensus.cc:2229] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:22.876924 29419 raft_consensus.cc:2258] T d150978500404c0c84d2c1be4767b06f P a10e98dae4fb4f52b56f05afd3f86313 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:22.975696 29419 tablet_server.cc:194] TabletServer@127.28.186.193:0 shutdown complete.
W20260806 20:14:22.977470 29682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.186.193:39817: connect: Connection refused (error 111)
W20260806 20:14:22.980971 29682 consensus_peers.cc:487] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 -> Peer a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817): Couldn't send request to peer a10e98dae4fb4f52b56f05afd3f86313. Status: Network error: Client connection negotiation failed: client connection to 127.28.186.193:39817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:22.993736 29419 tablet_server.cc:178] TabletServer@127.28.186.194:0 shutting down...
I20260806 20:14:23.019001 29419 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:14:23.019649 29419 tablet_replica.cc:330] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7: stopping tablet replica
I20260806 20:14:23.020212 29419 raft_consensus.cc:2229] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 4 LEADER]: Raft consensus shutting down.
I20260806 20:14:23.021050 29419 raft_consensus.cc:2258] T 3a8b884e76a547bf8ff5a7c371f629a1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.022805 29419 tablet_replica.cc:330] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7: stopping tablet replica
I20260806 20:14:23.023289 29419 raft_consensus.cc:2229] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.023720 29419 raft_consensus.cc:2258] T 2b08a45d83e140d7b6a07558f4da9a63 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.025375 29419 tablet_replica.cc:330] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7: stopping tablet replica
I20260806 20:14:23.025890 29419 raft_consensus.cc:2229] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.026342 29419 raft_consensus.cc:2258] T 8bd4523c8fcd4e3c950fe41d0a908f19 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.027899 29419 tablet_replica.cc:330] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7: stopping tablet replica
I20260806 20:14:23.028371 29419 raft_consensus.cc:2229] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:14:23.029147 29419 raft_consensus.cc:2258] T d150978500404c0c84d2c1be4767b06f P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.030794 29419 tablet_replica.cc:330] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7: stopping tablet replica
I20260806 20:14:23.031235 29419 raft_consensus.cc:2229] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.031622 29419 raft_consensus.cc:2258] T d133841d936c42e5a45fe986450328e3 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.033170 29419 tablet_replica.cc:330] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7: stopping tablet replica
I20260806 20:14:23.033671 29419 raft_consensus.cc:2229] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.034117 29419 raft_consensus.cc:2258] T 14689f97dadc4b0e95a25cedc9f172cc P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.035614 29419 tablet_replica.cc:330] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7: stopping tablet replica
I20260806 20:14:23.036042 29419 raft_consensus.cc:2229] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.036429 29419 raft_consensus.cc:2258] T f6e60d5891ab409ea2a360dd221d3d1e P d5ad1e1b5e874ccea53daf19b90f22a7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.038024 29419 tablet_replica.cc:330] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7: stopping tablet replica
I20260806 20:14:23.038453 29419 raft_consensus.cc:2229] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.038847 29419 raft_consensus.cc:2258] T 3474e05293044f6cbbf938d0f54c9dd1 P d5ad1e1b5e874ccea53daf19b90f22a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.040445 29419 tablet_replica.cc:330] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7: stopping tablet replica
I20260806 20:14:23.040867 29419 raft_consensus.cc:2229] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:14:23.041824 29419 raft_consensus.cc:2258] T b3ff2293355b476b85beb264fc24f12d P d5ad1e1b5e874ccea53daf19b90f22a7 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260806 20:14:23.045559 29682 consensus_peers.cc:487] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 -> Peer a10e98dae4fb4f52b56f05afd3f86313 (127.28.186.193:39817): Couldn't send request to peer a10e98dae4fb4f52b56f05afd3f86313. Status: Network error: Client connection negotiation failed: client connection to 127.28.186.193:39817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:23.071482 29419 tablet_server.cc:194] TabletServer@127.28.186.194:0 shutdown complete.
I20260806 20:14:23.088501 29419 tablet_server.cc:178] TabletServer@127.28.186.195:0 shutting down...
I20260806 20:14:23.111549 29419 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:14:23.112346 29419 tablet_replica.cc:330] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8: stopping tablet replica
I20260806 20:14:23.113003 29419 raft_consensus.cc:2229] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:14:23.113888 29419 raft_consensus.cc:2258] T 2b08a45d83e140d7b6a07558f4da9a63 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.115810 29419 tablet_replica.cc:330] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8: stopping tablet replica
W20260806 20:14:23.115969 29684 consensus_peers.cc:487] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 -> Peer d5ad1e1b5e874ccea53daf19b90f22a7 (127.28.186.194:45689): Couldn't send request to peer d5ad1e1b5e874ccea53daf19b90f22a7. Status: Network error: Client connection negotiation failed: client connection to 127.28.186.194:45689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:23.116375 29419 raft_consensus.cc:2229] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:14:23.117146 29419 raft_consensus.cc:2258] T 3474e05293044f6cbbf938d0f54c9dd1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.118808 29419 tablet_replica.cc:330] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8: stopping tablet replica
I20260806 20:14:23.119302 29419 raft_consensus.cc:2229] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.119693 29419 raft_consensus.cc:2258] T 3a8b884e76a547bf8ff5a7c371f629a1 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.121274 29419 tablet_replica.cc:330] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8: stopping tablet replica
I20260806 20:14:23.121764 29419 raft_consensus.cc:2229] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.122196 29419 raft_consensus.cc:2258] T 8bd4523c8fcd4e3c950fe41d0a908f19 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.123749 29419 tablet_replica.cc:330] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8: stopping tablet replica
I20260806 20:14:23.124221 29419 raft_consensus.cc:2229] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.124607 29419 raft_consensus.cc:2258] T d150978500404c0c84d2c1be4767b06f P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.126291 29419 tablet_replica.cc:330] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8: stopping tablet replica
I20260806 20:14:23.126802 29419 raft_consensus.cc:2229] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.127193 29419 raft_consensus.cc:2258] T d133841d936c42e5a45fe986450328e3 P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.128684 29419 tablet_replica.cc:330] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8: stopping tablet replica
I20260806 20:14:23.129140 29419 raft_consensus.cc:2229] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.129612 29419 raft_consensus.cc:2258] T 14689f97dadc4b0e95a25cedc9f172cc P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.131196 29419 tablet_replica.cc:330] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8: stopping tablet replica
I20260806 20:14:23.131654 29419 raft_consensus.cc:2229] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 4 LEADER]: Raft consensus shutting down.
I20260806 20:14:23.132402 29419 raft_consensus.cc:2258] T f6e60d5891ab409ea2a360dd221d3d1e P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.133965 29419 tablet_replica.cc:330] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8: stopping tablet replica
I20260806 20:14:23.134359 29419 raft_consensus.cc:2229] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:23.134768 29419 raft_consensus.cc:2258] T b3ff2293355b476b85beb264fc24f12d P 96cbcc4fe5f841d798d8b6131fafd3f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.161867 29419 tablet_server.cc:194] TabletServer@127.28.186.195:0 shutdown complete.
I20260806 20:14:23.179847 29419 master.cc:537] Master@127.28.186.254:41559 shutting down...
I20260806 20:14:23.404788 29419 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:14:23.405411 29419 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:23.405789 29419 tablet_replica.cc:330] T 00000000000000000000000000000000 P aceb4a2bf355451fa5aa3543394be475: stopping tablet replica
I20260806 20:14:23.425079 29419 master.cc:552] Master@127.28.186.254:41559 shutdown complete.
96cbcc4fe5f841d798d8b6131fafd3f8 1
a10e98dae4fb4f52b56f05afd3f86313 4
d5ad1e1b5e874ccea53daf19b90f22a7 4
96cbcc4fe5f841d798d8b6131fafd3f8 3
a10e98dae4fb4f52b56f05afd3f86313 3
d5ad1e1b5e874ccea53daf19b90f22a7 3
96cbcc4fe5f841d798d8b6131fafd3f8 1
a10e98dae4fb4f52b56f05afd3f86313 0
d5ad1e1b5e874ccea53daf19b90f22a7 8
96cbcc4fe5f841d798d8b6131fafd3f8 3
a10e98dae4fb4f52b56f05afd3f86313 3
d5ad1e1b5e874ccea53daf19b90f22a7 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31274 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260806 20:14:23.463763 29419 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.186.254:44285
I20260806 20:14:23.464821 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:14:23.470444 29974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:23.470630 29976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:23.471949 29419 server_base.cc:1034] running on GCE node
W20260806 20:14:23.472703 29978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:23.473665 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:14:23.473856 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:14:23.473964 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047263473956 us; error 0 us; skew 500 ppm
I20260806 20:14:23.474404 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:14:23.476807 29419 webserver.cc:460] Webserver started at http://127.28.186.254:34747/ using document root <none> and password file <none>
I20260806 20:14:23.477255 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:14:23.477406 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:14:23.477720 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:14:23.478881 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/master-0-root/instance:
uuid: "a432c5fe983f4d9d9723bfcbaccde844"
format_stamp: "Formatted at 2026-08-06 20:14:23 on dist-test-slave-49h3"
I20260806 20:14:23.483685 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 20:14:23.487093 29984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:23.487790 29419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 20:14:23.488029 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/master-0-root
uuid: "a432c5fe983f4d9d9723bfcbaccde844"
format_stamp: "Formatted at 2026-08-06 20:14:23 on dist-test-slave-49h3"
I20260806 20:14:23.488286 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:14:23.501137 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:14:23.502457 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:14:23.540091 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.254:44285
I20260806 20:14:23.540150 30036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.254:44285 every 8 connection(s)
I20260806 20:14:23.543848 30037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:23.553344 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844: Bootstrap starting.
I20260806 20:14:23.557870 30037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:23.561936 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844: No bootstrap required, opened a new log
I20260806 20:14:23.563900 30037 raft_consensus.cc:348] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a432c5fe983f4d9d9723bfcbaccde844" member_type: VOTER }
I20260806 20:14:23.564345 30037 raft_consensus.cc:374] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:23.564563 30037 raft_consensus.cc:729] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a432c5fe983f4d9d9723bfcbaccde844, State: Initialized, Role: FOLLOWER
I20260806 20:14:23.565090 30037 consensus_queue.cc:260] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a432c5fe983f4d9d9723bfcbaccde844" member_type: VOTER }
I20260806 20:14:23.565541 30037 raft_consensus.cc:388] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:14:23.565753 30037 raft_consensus.cc:482] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:14:23.566007 30037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:23.569829 30037 raft_consensus.cc:504] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a432c5fe983f4d9d9723bfcbaccde844" member_type: VOTER }
I20260806 20:14:23.570317 30037 leader_election.cc:302] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a432c5fe983f4d9d9723bfcbaccde844; no voters:
I20260806 20:14:23.571427 30037 leader_election.cc:288] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:14:23.571859 30040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:23.573127 30040 raft_consensus.cc:686] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 1 LEADER]: Becoming Leader. State: Replica: a432c5fe983f4d9d9723bfcbaccde844, State: Running, Role: LEADER
I20260806 20:14:23.573799 30040 consensus_queue.cc:237] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a432c5fe983f4d9d9723bfcbaccde844" member_type: VOTER }
I20260806 20:14:23.574210 30037 sys_catalog.cc:564] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:14:23.578346 30041 sys_catalog.cc:455] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a432c5fe983f4d9d9723bfcbaccde844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a432c5fe983f4d9d9723bfcbaccde844" member_type: VOTER } }
I20260806 20:14:23.579123 30041 sys_catalog.cc:458] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [sys.catalog]: This master's current role is: LEADER
I20260806 20:14:23.578820 30042 sys_catalog.cc:455] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a432c5fe983f4d9d9723bfcbaccde844. Latest consensus state: current_term: 1 leader_uuid: "a432c5fe983f4d9d9723bfcbaccde844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a432c5fe983f4d9d9723bfcbaccde844" member_type: VOTER } }
I20260806 20:14:23.580320 30042 sys_catalog.cc:458] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [sys.catalog]: This master's current role is: LEADER
I20260806 20:14:23.582448 30048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:14:23.587435 30048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:14:23.590968 29419 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 20:14:23.596474 30048 catalog_manager.cc:1347] Generated new cluster ID: 3534fa34993e480f9f434ec337a9e9e6
I20260806 20:14:23.596794 30048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:14:23.622112 30048 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:14:23.623684 30048 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:14:23.634222 30048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844: Generated new TSK 0
I20260806 20:14:23.634881 30048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:14:23.658121 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:14:23.663836 30060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:23.664992 30061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:23.667380 30063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:23.667541 29419 server_base.cc:1034] running on GCE node
I20260806 20:14:23.668572 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:14:23.668747 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:14:23.668861 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047263668853 us; error 0 us; skew 500 ppm
I20260806 20:14:23.669308 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:14:23.671650 29419 webserver.cc:460] Webserver started at http://127.28.186.193:42635/ using document root <none> and password file <none>
I20260806 20:14:23.672135 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:14:23.672297 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:14:23.672503 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:14:23.673612 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-0-root/instance:
uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
format_stamp: "Formatted at 2026-08-06 20:14:23 on dist-test-slave-49h3"
I20260806 20:14:23.678364 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 20:14:23.681758 30068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:23.682521 29419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 20:14:23.683183 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-0-root
uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
format_stamp: "Formatted at 2026-08-06 20:14:23 on dist-test-slave-49h3"
I20260806 20:14:23.683476 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:14:23.707023 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:14:23.708280 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:14:23.709815 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:14:23.712073 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:14:23.712271 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:23.712497 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:14:23.712653 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:23.751190 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.193:45001
I20260806 20:14:23.751274 30134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.193:45001 every 8 connection(s)
I20260806 20:14:23.755970 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:14:23.762893 30139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:23.763760 30140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:23.767057 29419 server_base.cc:1034] running on GCE node
W20260806 20:14:23.767805 30142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:23.768981 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:14:23.769223 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:14:23.769400 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047263769383 us; error 0 us; skew 500 ppm
I20260806 20:14:23.769950 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:14:23.772010 30135 heartbeater.cc:347] Connected to a master server at 127.28.186.254:44285
I20260806 20:14:23.772451 30135 heartbeater.cc:464] Registering TS with master...
I20260806 20:14:23.772655 29419 webserver.cc:460] Webserver started at http://127.28.186.194:35745/ using document root <none> and password file <none>
I20260806 20:14:23.773283 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:14:23.773279 30135 heartbeater.cc:511] Master 127.28.186.254:44285 requested a full tablet report, sending...
I20260806 20:14:23.773690 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:14:23.774035 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:14:23.775379 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-1-root/instance:
uuid: "975e4cda6abe46838b148df041cc3b2d"
format_stamp: "Formatted at 2026-08-06 20:14:23 on dist-test-slave-49h3"
I20260806 20:14:23.775810 30001 ts_manager.cc:194] Registered new tserver with Master: b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001)
I20260806 20:14:23.778453 30001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46762
I20260806 20:14:23.781383 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 20:14:23.785105 30148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:23.785969 29419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:14:23.786263 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-1-root
uuid: "975e4cda6abe46838b148df041cc3b2d"
format_stamp: "Formatted at 2026-08-06 20:14:23 on dist-test-slave-49h3"
I20260806 20:14:23.786505 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:14:23.808035 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:14:23.809226 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:14:23.810702 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:14:23.812939 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:14:23.813133 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:23.813359 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:14:23.813522 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 20:14:23.852118 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.194:36663
I20260806 20:14:23.852218 30215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.194:36663 every 8 connection(s)
I20260806 20:14:23.856802 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:14:23.863107 30219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:23.864351 30220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:23.867393 29419 server_base.cc:1034] running on GCE node
W20260806 20:14:23.867591 30222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:23.868568 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:14:23.868778 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:14:23.868950 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047263868936 us; error 0 us; skew 500 ppm
I20260806 20:14:23.869637 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:14:23.870579 30216 heartbeater.cc:347] Connected to a master server at 127.28.186.254:44285
I20260806 20:14:23.871013 30216 heartbeater.cc:464] Registering TS with master...
I20260806 20:14:23.871728 30216 heartbeater.cc:511] Master 127.28.186.254:44285 requested a full tablet report, sending...
I20260806 20:14:23.872469 29419 webserver.cc:460] Webserver started at http://127.28.186.195:35565/ using document root <none> and password file <none>
I20260806 20:14:23.873073 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:14:23.873287 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:14:23.873571 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:14:23.874017 30001 ts_manager.cc:194] Registered new tserver with Master: 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:23.874807 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-2-root/instance:
uuid: "6dad604353364122ae41c129a3c5cb46"
format_stamp: "Formatted at 2026-08-06 20:14:23 on dist-test-slave-49h3"
I20260806 20:14:23.875566 30001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46776
I20260806 20:14:23.879681 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 20:14:23.883102 30227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:23.883824 29419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 20:14:23.884124 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-2-root
uuid: "6dad604353364122ae41c129a3c5cb46"
format_stamp: "Formatted at 2026-08-06 20:14:23 on dist-test-slave-49h3"
I20260806 20:14:23.884428 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786047231998201-29419-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:14:23.906432 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:14:23.907596 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:14:23.909003 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:14:23.911362 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:14:23.911554 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:23.911769 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:14:23.911912 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:23.955084 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.195:33715
I20260806 20:14:23.955186 30294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.195:33715 every 8 connection(s)
I20260806 20:14:23.969810 30295 heartbeater.cc:347] Connected to a master server at 127.28.186.254:44285
I20260806 20:14:23.970218 30295 heartbeater.cc:464] Registering TS with master...
I20260806 20:14:23.970853 30295 heartbeater.cc:511] Master 127.28.186.254:44285 requested a full tablet report, sending...
I20260806 20:14:23.972708 30001 ts_manager.cc:194] Registered new tserver with Master: 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:23.972903 29419 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01466815s
I20260806 20:14:23.974550 30001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46782
I20260806 20:14:23.975630 29419 test_util.cc:268] Using random seed: -2001164104
I20260806 20:14:24.000774 30001 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46798:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:14:24.003688 30001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:14:24.103988 30099 tablet_service.cc:1467] Processing CreateTablet for tablet 7aec719ee2834a7ba6c7d822c788ad3e (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 20:14:24.105509 30099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aec719ee2834a7ba6c7d822c788ad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.106572 30095 tablet_service.cc:1467] Processing CreateTablet for tablet 78a4033e9b914dd4a00cbfeb3f6b7e66 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 20:14:24.108037 30095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a4033e9b914dd4a00cbfeb3f6b7e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.108778 30097 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6a183f6d714d0ba65ca608304b96d0 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 20:14:24.109249 30172 tablet_service.cc:1467] Processing CreateTablet for tablet 206cc589130d4950abb1a8cf15116498 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 20:14:24.109954 30097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6a183f6d714d0ba65ca608304b96d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.110612 30172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 206cc589130d4950abb1a8cf15116498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.111776 30093 tablet_service.cc:1467] Processing CreateTablet for tablet 4d187995bdd1494c89d80604e6f65944 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 20:14:24.113023 30093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d187995bdd1494c89d80604e6f65944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.110634 30094 tablet_service.cc:1467] Processing CreateTablet for tablet 206cc589130d4950abb1a8cf15116498 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 20:14:24.114776 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 206cc589130d4950abb1a8cf15116498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.114637 30092 tablet_service.cc:1467] Processing CreateTablet for tablet 898d35ff17224b77a542377b8b3fa397 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 20:14:24.116858 30173 tablet_service.cc:1467] Processing CreateTablet for tablet 78a4033e9b914dd4a00cbfeb3f6b7e66 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 20:14:24.118332 30173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a4033e9b914dd4a00cbfeb3f6b7e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.119464 30248 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef940c78b6f4978a1802a7fffe371db (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 20:14:24.120862 30248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef940c78b6f4978a1802a7fffe371db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.122676 30174 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6a183f6d714d0ba65ca608304b96d0 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 20:14:24.124020 30174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6a183f6d714d0ba65ca608304b96d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.126861 30175 tablet_service.cc:1467] Processing CreateTablet for tablet e48f8dee5c9d48498fc4a60b781686c2 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 20:14:24.128188 30175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48f8dee5c9d48498fc4a60b781686c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.130692 30176 tablet_service.cc:1467] Processing CreateTablet for tablet 7aec719ee2834a7ba6c7d822c788ad3e (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 20:14:24.131959 30176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aec719ee2834a7ba6c7d822c788ad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.134637 30171 tablet_service.cc:1467] Processing CreateTablet for tablet 4d187995bdd1494c89d80604e6f65944 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 20:14:24.135959 30171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d187995bdd1494c89d80604e6f65944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.138651 30170 tablet_service.cc:1467] Processing CreateTablet for tablet 898d35ff17224b77a542377b8b3fa397 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 20:14:24.139928 30170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898d35ff17224b77a542377b8b3fa397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.142657 30169 tablet_service.cc:1467] Processing CreateTablet for tablet 89914b5b49b143a98cfe92a893ef4569 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 20:14:24.144052 30169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89914b5b49b143a98cfe92a893ef4569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.146711 30168 tablet_service.cc:1467] Processing CreateTablet for tablet e99ec4f4772544638e34273d315381e8 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 20:14:24.154505 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6a183f6d714d0ba65ca608304b96d0 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 20:14:24.156038 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6a183f6d714d0ba65ca608304b96d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.158676 30252 tablet_service.cc:1467] Processing CreateTablet for tablet 4d187995bdd1494c89d80604e6f65944 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 20:14:24.160166 30252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d187995bdd1494c89d80604e6f65944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.162647 30254 tablet_service.cc:1467] Processing CreateTablet for tablet 78a4033e9b914dd4a00cbfeb3f6b7e66 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 20:14:24.164206 30254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a4033e9b914dd4a00cbfeb3f6b7e66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.166702 30253 tablet_service.cc:1467] Processing CreateTablet for tablet 206cc589130d4950abb1a8cf15116498 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 20:14:24.168151 30253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 206cc589130d4950abb1a8cf15116498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.170648 30251 tablet_service.cc:1467] Processing CreateTablet for tablet 898d35ff17224b77a542377b8b3fa397 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 20:14:24.172976 30091 tablet_service.cc:1467] Processing CreateTablet for tablet 89914b5b49b143a98cfe92a893ef4569 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 20:14:24.175168 30256 tablet_service.cc:1467] Processing CreateTablet for tablet e48f8dee5c9d48498fc4a60b781686c2 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 20:14:24.176502 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48f8dee5c9d48498fc4a60b781686c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.178560 30167 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef940c78b6f4978a1802a7fffe371db (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 20:14:24.179018 30251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898d35ff17224b77a542377b8b3fa397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.181869 30168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99ec4f4772544638e34273d315381e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.118324 30249 tablet_service.cc:1467] Processing CreateTablet for tablet e99ec4f4772544638e34273d315381e8 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 20:14:24.184684 30250 tablet_service.cc:1467] Processing CreateTablet for tablet 89914b5b49b143a98cfe92a893ef4569 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 20:14:24.182564 30092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 898d35ff17224b77a542377b8b3fa397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.188695 30313 tablet_bootstrap.cc:492] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.189038 30312 tablet_bootstrap.cc:492] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.188958 30098 tablet_service.cc:1467] Processing CreateTablet for tablet e48f8dee5c9d48498fc4a60b781686c2 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 20:14:24.189570 30091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89914b5b49b143a98cfe92a893ef4569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.180672 30257 tablet_service.cc:1467] Processing CreateTablet for tablet 7aec719ee2834a7ba6c7d822c788ad3e (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 20:14:24.190953 30167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef940c78b6f4978a1802a7fffe371db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.192965 30098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48f8dee5c9d48498fc4a60b781686c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.193706 30249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99ec4f4772544638e34273d315381e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.194303 30250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89914b5b49b143a98cfe92a893ef4569. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.197160 30097 tablet_service.cc:1467] Processing CreateTablet for tablet e99ec4f4772544638e34273d315381e8 (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 20:14:24.198488 30097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99ec4f4772544638e34273d315381e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.201865 30257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aec719ee2834a7ba6c7d822c788ad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.218807 30312 tablet_bootstrap.cc:654] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.222352 30094 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef940c78b6f4978a1802a7fffe371db (DEFAULT_TABLE table=test-workload [id=c1da50ea48e54f28b47489354f251ef6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 20:14:24.223649 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef940c78b6f4978a1802a7fffe371db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:24.257030 30314 tablet_bootstrap.cc:492] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.268724 30313 tablet_bootstrap.cc:654] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.290450 30312 tablet_bootstrap.cc:492] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.290947 30312 ts_tablet_manager.cc:1397] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.102s user 0.013s sys 0.027s
I20260806 20:14:24.293601 30312 raft_consensus.cc:348] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.294335 30312 raft_consensus.cc:374] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.294617 30312 raft_consensus.cc:729] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.295270 30312 consensus_queue.cc:260] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.296689 30314 tablet_bootstrap.cc:654] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.300206 30135 heartbeater.cc:503] Master 127.28.186.254:44285 was elected leader, sending a full tablet report...
I20260806 20:14:24.300454 30312 ts_tablet_manager.cc:1428] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.009s user 0.007s sys 0.003s
I20260806 20:14:24.301306 30312 tablet_bootstrap.cc:492] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.306345 30312 tablet_bootstrap.cc:654] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.311518 30312 tablet_bootstrap.cc:492] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.311962 30313 tablet_bootstrap.cc:492] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.312625 30313 ts_tablet_manager.cc:1397] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.008s
I20260806 20:14:24.312008 30312 ts_tablet_manager.cc:1397] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 20:14:24.314872 30314 tablet_bootstrap.cc:492] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.315358 30314 ts_tablet_manager.cc:1397] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.059s user 0.007s sys 0.006s
I20260806 20:14:24.315400 30312 raft_consensus.cc:348] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.315770 30313 raft_consensus.cc:348] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.316262 30312 raft_consensus.cc:374] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.316658 30313 raft_consensus.cc:374] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.316694 30312 raft_consensus.cc:729] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.317196 30313 raft_consensus.cc:729] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.317608 30312 consensus_queue.cc:260] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.318120 30314 raft_consensus.cc:348] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.318059 30313 consensus_queue.cc:260] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.319044 30314 raft_consensus.cc:374] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.319366 30314 raft_consensus.cc:729] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.320083 30312 ts_tablet_manager.cc:1428] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:14:24.320108 30314 consensus_queue.cc:260] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.321189 30312 tablet_bootstrap.cc:492] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.327096 30295 heartbeater.cc:503] Master 127.28.186.254:44285 was elected leader, sending a full tablet report...
I20260806 20:14:24.327922 30312 tablet_bootstrap.cc:654] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.332460 30313 ts_tablet_manager.cc:1428] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.019s user 0.008s sys 0.004s
I20260806 20:14:24.333521 30313 tablet_bootstrap.cc:492] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.333714 30312 tablet_bootstrap.cc:492] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.334196 30312 ts_tablet_manager.cc:1397] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260806 20:14:24.334864 30314 ts_tablet_manager.cc:1428] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.019s user 0.009s sys 0.008s
I20260806 20:14:24.335587 30216 heartbeater.cc:503] Master 127.28.186.254:44285 was elected leader, sending a full tablet report...
I20260806 20:14:24.336789 30312 raft_consensus.cc:348] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.337548 30312 raft_consensus.cc:374] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.337834 30312 raft_consensus.cc:729] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.338640 30312 consensus_queue.cc:260] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.340626 30313 tablet_bootstrap.cc:654] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.344604 30314 tablet_bootstrap.cc:492] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.344712 30312 ts_tablet_manager.cc:1428] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20260806 20:14:24.345707 30312 tablet_bootstrap.cc:492] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.351261 30312 tablet_bootstrap.cc:654] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.351256 30314 tablet_bootstrap.cc:654] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.353518 30321 raft_consensus.cc:482] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.354084 30321 raft_consensus.cc:504] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.358458 30321 leader_election.cc:288] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.366856 30312 tablet_bootstrap.cc:492] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.367583 30312 ts_tablet_manager.cc:1397] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.011s
I20260806 20:14:24.370605 30312 raft_consensus.cc:348] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.371635 30312 raft_consensus.cc:374] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.372009 30312 raft_consensus.cc:729] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.372839 30312 consensus_queue.cc:260] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.375470 30312 ts_tablet_manager.cc:1428] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:14:24.378407 30312 tablet_bootstrap.cc:492] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.380666 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206cc589130d4950abb1a8cf15116498" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.383612 30316 raft_consensus.cc:482] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.384088 30316 raft_consensus.cc:504] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.386401 30314 tablet_bootstrap.cc:492] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.386873 30314 ts_tablet_manager.cc:1397] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.006s
I20260806 20:14:24.387504 30312 tablet_bootstrap.cc:654] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.386616 30316 leader_election.cc:288] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.389262 30313 tablet_bootstrap.cc:492] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.394397 30313 ts_tablet_manager.cc:1397] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.061s user 0.013s sys 0.013s
I20260806 20:14:24.383232 30109 raft_consensus.cc:2454] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 0.
I20260806 20:14:24.400075 30313 raft_consensus.cc:348] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.403954 30313 raft_consensus.cc:374] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.404318 30313 raft_consensus.cc:729] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.400027 30314 raft_consensus.cc:348] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.405103 30313 consensus_queue.cc:260] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.407968 30314 raft_consensus.cc:374] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.409701 30313 ts_tablet_manager.cc:1428] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20260806 20:14:24.408346 30314 raft_consensus.cc:729] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.410616 30313 tablet_bootstrap.cc:492] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.411196 30314 consensus_queue.cc:260] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.414175 30312 tablet_bootstrap.cc:492] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.415827 30312 ts_tablet_manager.cc:1397] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.004s
I20260806 20:14:24.415920 30314 ts_tablet_manager.cc:1428] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260806 20:14:24.416882 30314 tablet_bootstrap.cc:492] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.416919 30313 tablet_bootstrap.cc:654] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.418725 30312 raft_consensus.cc:348] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.419543 30312 raft_consensus.cc:374] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.420049 30312 raft_consensus.cc:729] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.420832 30312 consensus_queue.cc:260] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.422266 30150 leader_election.cc:302] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.423161 30321 raft_consensus.cc:2781] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.423615 30321 raft_consensus.cc:482] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.423934 30321 raft_consensus.cc:3037] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.428601 30314 tablet_bootstrap.cc:654] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.423161 30312 ts_tablet_manager.cc:1428] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:14:24.431216 30312 tablet_bootstrap.cc:492] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.432019 30267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206cc589130d4950abb1a8cf15116498" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:24.432745 30267 raft_consensus.cc:2454] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 0.
I20260806 20:14:24.432679 30321 raft_consensus.cc:504] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.435459 30321 leader_election.cc:288] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.435578 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206cc589130d4950abb1a8cf15116498" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:24.436164 30109 raft_consensus.cc:3037] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.437762 30312 tablet_bootstrap.cc:654] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.442268 30267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206cc589130d4950abb1a8cf15116498" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:24.442870 30267 raft_consensus.cc:3037] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.444267 30109 raft_consensus.cc:2454] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 1.
I20260806 20:14:24.445303 30150 leader_election.cc:302] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.446206 30313 tablet_bootstrap.cc:492] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.446215 30321 raft_consensus.cc:2781] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:24.446836 30313 ts_tablet_manager.cc:1397] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.008s
I20260806 20:14:24.451795 30316 raft_consensus.cc:482] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.452292 30316 raft_consensus.cc:504] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.454615 30316 leader_election.cc:288] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.458416 30267 raft_consensus.cc:2454] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 1.
I20260806 20:14:24.452718 30313 raft_consensus.cc:348] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.465484 30313 raft_consensus.cc:374] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.465792 30313 raft_consensus.cc:729] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.466440 30313 consensus_queue.cc:260] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.468144 30313 ts_tablet_manager.cc:1428] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20260806 20:14:24.469029 30313 tablet_bootstrap.cc:492] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.472193 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a183f6d714d0ba65ca608304b96d0" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:24.474638 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.475752 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a183f6d714d0ba65ca608304b96d0" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.475111 30267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:24.476435 30186 raft_consensus.cc:2454] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 0.
W20260806 20:14:24.476861 30070 leader_election.cc:341] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:24.475135 30321 raft_consensus.cc:686] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Becoming Leader. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:24.477712 30070 leader_election.cc:302] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.478552 30316 raft_consensus.cc:2781] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.478797 30312 tablet_bootstrap.cc:492] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.478288 30321 consensus_queue.cc:237] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.479300 30312 ts_tablet_manager.cc:1397] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.018s
W20260806 20:14:24.479581 30072 leader_election.cc:341] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:24.480022 30072 leader_election.cc:302] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b68c081f3ce2459b8d81dc29e3d4620d; no voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:24.480490 30313 tablet_bootstrap.cc:654] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.480916 30316 raft_consensus.cc:482] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.481272 30316 raft_consensus.cc:3037] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.482157 30312 raft_consensus.cc:348] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.483111 30312 raft_consensus.cc:374] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.483420 30312 raft_consensus.cc:729] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.484782 30312 consensus_queue.cc:260] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
W20260806 20:14:24.485292 30072 leader_election.cc:341] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:24.487321 30336 raft_consensus.cc:2726] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:24.492342 30314 tablet_bootstrap.cc:492] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.492919 30314 ts_tablet_manager.cc:1397] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.005s
I20260806 20:14:24.501880 30313 tablet_bootstrap.cc:492] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.502007 30316 raft_consensus.cc:504] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.501395 30314 raft_consensus.cc:348] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.502532 30313 ts_tablet_manager.cc:1397] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.010s
I20260806 20:14:24.502880 30314 raft_consensus.cc:374] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.503239 30314 raft_consensus.cc:729] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.504397 30316 leader_election.cc:288] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.504820 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a183f6d714d0ba65ca608304b96d0" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:24.505415 30186 raft_consensus.cc:3037] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.505561 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a183f6d714d0ba65ca608304b96d0" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
W20260806 20:14:24.507216 30072 leader_election.cc:341] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:24.508569 30313 raft_consensus.cc:348] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.509224 30313 raft_consensus.cc:374] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.509528 30313 raft_consensus.cc:729] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.510201 30313 consensus_queue.cc:260] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.513072 30186 raft_consensus.cc:2454] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 1.
I20260806 20:14:24.513412 30000 catalog_manager.cc:5495] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 975e4cda6abe46838b148df041cc3b2d (127.28.186.194). New cstate: current_term: 1 leader_uuid: "975e4cda6abe46838b148df041cc3b2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:24.514204 30313 ts_tablet_manager.cc:1428] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260806 20:14:24.514189 30070 leader_election.cc:302] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters: 6dad604353364122ae41c129a3c5cb46
I20260806 20:14:24.515388 30313 tablet_bootstrap.cc:492] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.515453 30316 raft_consensus.cc:2781] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:24.516004 30316 raft_consensus.cc:686] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:24.516783 30316 consensus_queue.cc:237] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.518890 30312 ts_tablet_manager.cc:1428] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.039s user 0.005s sys 0.005s
I20260806 20:14:24.520697 30312 tablet_bootstrap.cc:492] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.519248 30314 consensus_queue.cc:260] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.526854 30312 tablet_bootstrap.cc:654] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.527125 30314 ts_tablet_manager.cc:1428] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.033s user 0.001s sys 0.008s
I20260806 20:14:24.527910 30313 tablet_bootstrap.cc:654] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.530184 30314 tablet_bootstrap.cc:492] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.536199 30314 tablet_bootstrap.cc:654] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.541290 30312 tablet_bootstrap.cc:492] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.541891 30312 ts_tablet_manager.cc:1397] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.007s
I20260806 20:14:24.542393 30313 tablet_bootstrap.cc:492] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.542863 30313 ts_tablet_manager.cc:1397] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.004s
I20260806 20:14:24.545349 30313 raft_consensus.cc:348] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.546067 30313 raft_consensus.cc:374] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.546341 30313 raft_consensus.cc:729] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.546918 30312 raft_consensus.cc:348] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.547071 30313 consensus_queue.cc:260] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.547729 30312 raft_consensus.cc:374] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.548077 30312 raft_consensus.cc:729] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.547857 30000 catalog_manager.cc:5495] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 0 to 1, leader changed from <none> to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 1 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:24.548875 30312 consensus_queue.cc:260] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.551616 30312 ts_tablet_manager.cc:1428] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260806 20:14:24.552632 30312 tablet_bootstrap.cc:492] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.555891 30313 ts_tablet_manager.cc:1428] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260806 20:14:24.557760 30316 raft_consensus.cc:482] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.558228 30316 raft_consensus.cc:504] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.559803 30312 tablet_bootstrap.cc:654] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.560180 30316 leader_election.cc:288] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.561843 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.562408 30313 tablet_bootstrap.cc:492] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.562947 30314 tablet_bootstrap.cc:492] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.563453 30314 ts_tablet_manager.cc:1397] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.013s
I20260806 20:14:24.563618 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
W20260806 20:14:24.563958 30070 leader_election.cc:341] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:14:24.564889 30072 leader_election.cc:341] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:24.565302 30072 leader_election.cc:302] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b68c081f3ce2459b8d81dc29e3d4620d; no voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:24.566126 30314 raft_consensus.cc:348] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.566397 30316 raft_consensus.cc:2726] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:24.566648 30314 raft_consensus.cc:374] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.566978 30314 raft_consensus.cc:729] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.567631 30314 consensus_queue.cc:260] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.568162 30312 tablet_bootstrap.cc:492] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.568611 30312 ts_tablet_manager.cc:1397] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260806 20:14:24.569855 30314 ts_tablet_manager.cc:1428] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260806 20:14:24.570744 30314 tablet_bootstrap.cc:492] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.571339 30312 raft_consensus.cc:348] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.571987 30312 raft_consensus.cc:374] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.572237 30312 raft_consensus.cc:729] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.572862 30312 consensus_queue.cc:260] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.574767 30312 ts_tablet_manager.cc:1428] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260806 20:14:24.575872 30312 tablet_bootstrap.cc:492] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.576747 30314 tablet_bootstrap.cc:654] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.581102 30312 tablet_bootstrap.cc:654] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
W20260806 20:14:24.586136 30050 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:14:24.588989 30313 tablet_bootstrap.cc:654] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.589581 30314 tablet_bootstrap.cc:492] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.590103 30314 ts_tablet_manager.cc:1397] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20260806 20:14:24.592895 30314 raft_consensus.cc:348] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.593693 30314 raft_consensus.cc:374] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.593977 30314 raft_consensus.cc:729] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.595870 30314 consensus_queue.cc:260] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.596932 30312 tablet_bootstrap.cc:492] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.597368 30312 ts_tablet_manager.cc:1397] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.012s
I20260806 20:14:24.598294 30314 ts_tablet_manager.cc:1428] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 20:14:24.599452 30314 tablet_bootstrap.cc:492] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.599628 30312 raft_consensus.cc:348] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.600278 30312 raft_consensus.cc:374] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.600521 30312 raft_consensus.cc:729] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.601105 30312 consensus_queue.cc:260] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.605340 30312 ts_tablet_manager.cc:1428] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260806 20:14:24.605644 30314 tablet_bootstrap.cc:654] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.606284 30312 tablet_bootstrap.cc:492] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d: Bootstrap starting.
I20260806 20:14:24.611796 30312 tablet_bootstrap.cc:654] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.613559 30313 tablet_bootstrap.cc:492] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.613791 30314 tablet_bootstrap.cc:492] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.614190 30313 ts_tablet_manager.cc:1397] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.052s user 0.021s sys 0.008s
I20260806 20:14:24.614293 30314 ts_tablet_manager.cc:1397] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260806 20:14:24.617285 30314 raft_consensus.cc:348] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.618048 30314 raft_consensus.cc:374] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.618068 30313 raft_consensus.cc:348] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.618528 30314 raft_consensus.cc:729] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.618765 30312 tablet_bootstrap.cc:492] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d: No bootstrap required, opened a new log
I20260806 20:14:24.619191 30312 ts_tablet_manager.cc:1397] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260806 20:14:24.619316 30314 consensus_queue.cc:260] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.621572 30314 ts_tablet_manager.cc:1428] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260806 20:14:24.621507 30312 raft_consensus.cc:348] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.622308 30312 raft_consensus.cc:374] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.622570 30312 raft_consensus.cc:729] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.622816 30313 raft_consensus.cc:374] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.623112 30313 raft_consensus.cc:729] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.623322 30312 consensus_queue.cc:260] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.623803 30313 consensus_queue.cc:260] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.625432 30312 ts_tablet_manager.cc:1428] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 20:14:24.625895 30313 ts_tablet_manager.cc:1428] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260806 20:14:24.626621 30313 tablet_bootstrap.cc:492] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.622553 30314 tablet_bootstrap.cc:492] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.635262 30314 tablet_bootstrap.cc:654] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.639766 30313 tablet_bootstrap.cc:654] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.640257 30314 tablet_bootstrap.cc:492] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.640713 30314 ts_tablet_manager.cc:1397] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20260806 20:14:24.643390 30314 raft_consensus.cc:348] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.643988 30314 raft_consensus.cc:374] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.644274 30314 raft_consensus.cc:729] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.645241 30314 consensus_queue.cc:260] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.645776 30313 tablet_bootstrap.cc:492] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.646339 30313 ts_tablet_manager.cc:1397] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20260806 20:14:24.647579 30314 ts_tablet_manager.cc:1428] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 20:14:24.648537 30314 tablet_bootstrap.cc:492] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.648890 30313 raft_consensus.cc:348] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.649657 30313 raft_consensus.cc:374] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.649932 30313 raft_consensus.cc:729] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.650625 30313 consensus_queue.cc:260] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.652897 30313 ts_tablet_manager.cc:1428] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 20:14:24.653842 30313 tablet_bootstrap.cc:492] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.654786 30314 tablet_bootstrap.cc:654] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.659418 30313 tablet_bootstrap.cc:654] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.660421 30314 tablet_bootstrap.cc:492] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.660902 30314 ts_tablet_manager.cc:1397] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260806 20:14:24.663313 30314 raft_consensus.cc:348] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.663938 30316 raft_consensus.cc:482] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.664004 30314 raft_consensus.cc:374] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.664412 30314 raft_consensus.cc:729] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.664764 30313 tablet_bootstrap.cc:492] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.664438 30316 raft_consensus.cc:504] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.665191 30313 ts_tablet_manager.cc:1397] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260806 20:14:24.665140 30314 consensus_queue.cc:260] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.666424 30316 leader_election.cc:288] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.667268 30314 ts_tablet_manager.cc:1428] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260806 20:14:24.667846 30313 raft_consensus.cc:348] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.668247 30314 tablet_bootstrap.cc:492] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.668494 30313 raft_consensus.cc:374] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.668467 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:24.668901 30313 raft_consensus.cc:729] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.669710 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.669570 30313 consensus_queue.cc:260] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.670149 30186 raft_consensus.cc:2454] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 0.
I20260806 20:14:24.671501 30313 ts_tablet_manager.cc:1428] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260806 20:14:24.672534 30313 tablet_bootstrap.cc:492] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.671514 30070 leader_election.cc:302] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.673894 30314 tablet_bootstrap.cc:654] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
W20260806 20:14:24.676484 30072 leader_election.cc:341] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:24.677609 30316 raft_consensus.cc:2781] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.677924 30313 tablet_bootstrap.cc:654] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.677986 30316 raft_consensus.cc:482] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.678503 30316 raft_consensus.cc:3037] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.682000 30320 raft_consensus.cc:482] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.682536 30320 raft_consensus.cc:504] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.685282 30320 leader_election.cc:288] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.697005 30316 raft_consensus.cc:504] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.706135 30339 raft_consensus.cc:482] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.706604 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:24.707254 30186 raft_consensus.cc:3037] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.706776 30339 raft_consensus.cc:504] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.710888 30339 leader_election.cc:288] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.711995 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99ec4f4772544638e34273d315381e8" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:24.712630 30266 raft_consensus.cc:2454] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 0.
I20260806 20:14:24.713867 30072 leader_election.cc:302] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.714253 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99ec4f4772544638e34273d315381e8" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.714825 30187 raft_consensus.cc:2454] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 0.
I20260806 20:14:24.717716 30339 raft_consensus.cc:2781] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.718176 30339 raft_consensus.cc:482] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.718541 30339 raft_consensus.cc:3037] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.720762 30186 raft_consensus.cc:2454] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 1.
I20260806 20:14:24.721909 30070 leader_election.cc:302] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.722736 30336 raft_consensus.cc:2781] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:24.727185 30336 raft_consensus.cc:686] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:24.728045 30336 consensus_queue.cc:237] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.730206 30339 raft_consensus.cc:504] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.734383 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:24.732343 30316 leader_election.cc:288] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
W20260806 20:14:24.736186 30072 leader_election.cc:341] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:24.734632 30339 leader_election.cc:288] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.738631 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99ec4f4772544638e34273d315381e8" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:24.739303 30266 raft_consensus.cc:3037] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.745870 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99ec4f4772544638e34273d315381e8" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:24.746675 30186 raft_consensus.cc:3037] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.759254 30266 raft_consensus.cc:2454] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 1.
I20260806 20:14:24.760461 30072 leader_election.cc:302] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.761937 30339 raft_consensus.cc:2781] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:24.762758 30339 raft_consensus.cc:686] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:24.763686 30339 consensus_queue.cc:237] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.766673 30186 raft_consensus.cc:2454] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 1.
I20260806 20:14:24.769394 30320 raft_consensus.cc:482] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.770399 30320 raft_consensus.cc:504] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.773301 30313 tablet_bootstrap.cc:492] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.773350 30320 leader_election.cc:288] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.773763 30313 ts_tablet_manager.cc:1397] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.101s user 0.024s sys 0.058s
I20260806 20:14:24.773932 30320 raft_consensus.cc:482] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.776288 30313 raft_consensus.cc:348] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.776971 30313 raft_consensus.cc:374] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.777227 30313 raft_consensus.cc:729] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.777535 30320 raft_consensus.cc:504] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.777884 30313 consensus_queue.cc:260] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.779385 30313 ts_tablet_manager.cc:1428] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260806 20:14:24.779462 30320 leader_election.cc:288] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.780189 30313 tablet_bootstrap.cc:492] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46: Bootstrap starting.
I20260806 20:14:24.779865 29998 catalog_manager.cc:5495] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 0 to 1, leader changed from <none> to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 1 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:24.782279 30336 raft_consensus.cc:482] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.784782 30336 raft_consensus.cc:504] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.784518 30320 raft_consensus.cc:482] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.791170 30314 tablet_bootstrap.cc:492] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.791616 30314 ts_tablet_manager.cc:1397] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.124s user 0.028s sys 0.062s
I20260806 20:14:24.793998 30313 tablet_bootstrap.cc:654] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.796123 30314 raft_consensus.cc:348] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.796844 30314 raft_consensus.cc:374] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.797086 30314 raft_consensus.cc:729] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.803666 30314 consensus_queue.cc:260] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.806362 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.807126 30186 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 0.
I20260806 20:14:24.808743 30070 leader_election.cc:302] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.809754 30339 raft_consensus.cc:2781] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.810125 30339 raft_consensus.cc:482] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.788645 30320 raft_consensus.cc:504] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.814059 30314 ts_tablet_manager.cc:1428] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260806 20:14:24.815464 30314 tablet_bootstrap.cc:492] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d: Bootstrap starting.
I20260806 20:14:24.812434 30339 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.817404 30336 leader_election.cc:288] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.818498 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:24.818939 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.819221 30266 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 0.
I20260806 20:14:24.819561 30186 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 0.
I20260806 20:14:24.824178 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.824862 30229 leader_election.cc:302] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:24.825291 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a183f6d714d0ba65ca608304b96d0" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.825986 30361 raft_consensus.cc:2781] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.826149 30187 raft_consensus.cc:2379] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6dad604353364122ae41c129a3c5cb46 in current term 1: Already voted for candidate b68c081f3ce2459b8d81dc29e3d4620d in this term.
I20260806 20:14:24.826402 30361 raft_consensus.cc:482] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.826732 30361 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.827229 30108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.827898 30108 raft_consensus.cc:2454] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 0.
I20260806 20:14:24.829262 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6a183f6d714d0ba65ca608304b96d0" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.829814 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.830431 30186 raft_consensus.cc:2454] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 0.
I20260806 20:14:24.831982 30320 leader_election.cc:288] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.833427 30339 raft_consensus.cc:504] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.834146 30361 raft_consensus.cc:504] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.834591 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.835187 30107 raft_consensus.cc:2454] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 0.
I20260806 20:14:24.834798 30109 raft_consensus.cc:2379] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6dad604353364122ae41c129a3c5cb46 in current term 1: Already voted for candidate b68c081f3ce2459b8d81dc29e3d4620d in this term.
I20260806 20:14:24.836073 30361 leader_election.cc:288] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.837695 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.838353 30187 raft_consensus.cc:2454] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 0.
I20260806 20:14:24.838536 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:24.836437 30339 leader_election.cc:288] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.839785 30229 leader_election.cc:302] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.840142 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:24.840654 30186 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.840981 30229 leader_election.cc:302] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.841874 30313 tablet_bootstrap.cc:492] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46: No bootstrap required, opened a new log
I20260806 20:14:24.839773 30266 raft_consensus.cc:2379] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b68c081f3ce2459b8d81dc29e3d4620d in current term 1: Already voted for candidate 6dad604353364122ae41c129a3c5cb46 in this term.
I20260806 20:14:24.840638 30314 tablet_bootstrap.cc:654] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:24.842726 30320 raft_consensus.cc:2781] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.843067 30320 raft_consensus.cc:482] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.843398 30320 raft_consensus.cc:3037] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.842401 30000 catalog_manager.cc:5495] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 0 to 1, leader changed from <none> to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 1 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:24.842335 30313 ts_tablet_manager.cc:1397] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46: Time spent bootstrapping tablet: real 0.062s user 0.020s sys 0.011s
I20260806 20:14:24.848703 30186 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 1.
I20260806 20:14:24.848685 30313 raft_consensus.cc:348] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.842543 30229 leader_election.cc:302] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46; no voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:24.840662 30361 raft_consensus.cc:2781] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.850147 30361 raft_consensus.cc:482] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.850487 30361 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.852253 30313 raft_consensus.cc:374] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.852548 30313 raft_consensus.cc:729] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.853289 30313 consensus_queue.cc:260] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.858938 30320 raft_consensus.cc:504] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.860101 30070 leader_election.cc:302] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters: 6dad604353364122ae41c129a3c5cb46
I20260806 20:14:24.861066 30320 leader_election.cc:288] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.861351 30321 raft_consensus.cc:482] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.861877 30321 raft_consensus.cc:504] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.863842 30321 leader_election.cc:288] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.864465 30339 raft_consensus.cc:2781] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:24.864969 30339 raft_consensus.cc:686] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:24.865929 30339 consensus_queue.cc:237] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.867003 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:24.869287 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:24.870177 30186 raft_consensus.cc:2379] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dad604353364122ae41c129a3c5cb46 in current term 1: Already voted for candidate b68c081f3ce2459b8d81dc29e3d4620d in this term.
I20260806 20:14:24.872371 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:24.873066 30186 raft_consensus.cc:3037] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.873817 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:24.874459 30109 raft_consensus.cc:3037] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.878718 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.879467 30107 raft_consensus.cc:2454] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 0.
I20260806 20:14:24.881228 30150 leader_election.cc:302] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.882369 30321 raft_consensus.cc:2781] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.882797 30321 raft_consensus.cc:482] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.883322 30321 raft_consensus.cc:3037] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.884641 30320 raft_consensus.cc:482] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.885586 30320 raft_consensus.cc:504] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.887007 30186 raft_consensus.cc:2454] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 1.
I20260806 20:14:24.888932 30368 raft_consensus.cc:482] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.889410 30368 raft_consensus.cc:504] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.892608 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.893594 29998 catalog_manager.cc:5495] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 0 to 1, leader changed from <none> to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 1 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:24.894630 30229 leader_election.cc:302] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:24.896378 30229 leader_election.cc:302] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46; no voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:24.896454 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:24.897202 30266 raft_consensus.cc:2454] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 0.
I20260806 20:14:24.900573 30366 raft_consensus.cc:2781] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:24.857054 30361 raft_consensus.cc:504] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.903995 30361 leader_election.cc:288] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.905032 30109 raft_consensus.cc:2454] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 1.
I20260806 20:14:24.906622 30313 ts_tablet_manager.cc:1428] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46: Time spent starting tablet: real 0.060s user 0.025s sys 0.015s
I20260806 20:14:24.909554 30361 raft_consensus.cc:2726] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:14:24.912220 30366 raft_consensus.cc:686] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Becoming Leader. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:24.864495 30365 raft_consensus.cc:3037] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.908131 30368 leader_election.cc:288] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.914467 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:24.915222 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:24.915562 30186 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.916312 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.915874 30109 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.917331 30107 raft_consensus.cc:2454] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 0.
I20260806 20:14:24.917860 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.918614 30187 raft_consensus.cc:2379] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6dad604353364122ae41c129a3c5cb46 in current term 1: Already voted for candidate b68c081f3ce2459b8d81dc29e3d4620d in this term.
I20260806 20:14:24.919713 30229 leader_election.cc:302] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.920025 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.920727 30368 raft_consensus.cc:2781] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.920795 30229 leader_election.cc:302] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46; no voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:24.921903 30370 raft_consensus.cc:3037] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.921159 30368 raft_consensus.cc:482] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.923236 30314 tablet_bootstrap.cc:492] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d: No bootstrap required, opened a new log
I20260806 20:14:24.923666 30314 ts_tablet_manager.cc:1397] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d: Time spent bootstrapping tablet: real 0.108s user 0.040s sys 0.027s
I20260806 20:14:24.915170 30320 leader_election.cc:288] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.924683 30365 raft_consensus.cc:2726] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:24.924414 30368 raft_consensus.cc:3037] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.928107 30314 raft_consensus.cc:348] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.928802 30314 raft_consensus.cc:374] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:24.929008 30314 raft_consensus.cc:729] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Initialized, Role: FOLLOWER
I20260806 20:14:24.929657 30314 consensus_queue.cc:260] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.930068 30333 raft_consensus.cc:482] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.930496 30333 raft_consensus.cc:504] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.932853 30333 leader_election.cc:288] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.936288 30186 raft_consensus.cc:2454] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 1.
I20260806 20:14:24.934188 30109 raft_consensus.cc:2454] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 1.
I20260806 20:14:24.935107 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.937495 30107 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 0.
I20260806 20:14:24.937942 30321 raft_consensus.cc:504] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.940040 30229 leader_election.cc:302] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.940742 30321 leader_election.cc:288] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.941489 30367 raft_consensus.cc:2781] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:24.942608 30187 raft_consensus.cc:2379] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6dad604353364122ae41c129a3c5cb46 in current term 1: Already voted for candidate 975e4cda6abe46838b148df041cc3b2d in this term.
I20260806 20:14:24.946152 30367 raft_consensus.cc:686] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Becoming Leader. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:24.947223 30314 ts_tablet_manager.cc:1428] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d: Time spent starting tablet: real 0.023s user 0.016s sys 0.004s
I20260806 20:14:24.942884 30150 leader_election.cc:302] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.947047 30367 consensus_queue.cc:237] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.944204 30368 raft_consensus.cc:504] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.948911 30321 raft_consensus.cc:2781] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:24.949296 30321 raft_consensus.cc:482] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:24.949478 30368 leader_election.cc:288] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.949682 30321 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.951112 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:24.951354 30267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:24.952103 30267 raft_consensus.cc:2379] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 975e4cda6abe46838b148df041cc3b2d in current term 1: Already voted for candidate 6dad604353364122ae41c129a3c5cb46 in this term.
I20260806 20:14:24.953696 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:24.954193 30107 raft_consensus.cc:3037] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.945329 30370 raft_consensus.cc:2726] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:24.929559 30365 raft_consensus.cc:482] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:24.927517 30366 consensus_queue.cc:237] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.954387 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:24.955569 30187 raft_consensus.cc:2379] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6dad604353364122ae41c129a3c5cb46 in current term 1: Already voted for candidate 975e4cda6abe46838b148df041cc3b2d in this term.
I20260806 20:14:24.957612 30365 raft_consensus.cc:504] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.961851 30107 raft_consensus.cc:2454] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 1.
I20260806 20:14:24.962538 30266 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 0.
I20260806 20:14:24.961884 30321 raft_consensus.cc:504] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.965027 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:24.969791 30109 raft_consensus.cc:2379] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 975e4cda6abe46838b148df041cc3b2d in current term 1: Already voted for candidate 6dad604353364122ae41c129a3c5cb46 in this term.
I20260806 20:14:24.970216 30321 leader_election.cc:288] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:24.967782 30365 leader_election.cc:288] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:24.971012 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:24.971576 30107 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.971549 30150 leader_election.cc:302] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d; no voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:24.972870 30321 raft_consensus.cc:2726] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:14:24.968998 30229 leader_election.cc:302] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters: 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:24.974344 30367 raft_consensus.cc:2781] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:24.974787 30367 raft_consensus.cc:686] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Becoming Leader. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:24.974012 29998 catalog_manager.cc:5495] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dad604353364122ae41c129a3c5cb46 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "6dad604353364122ae41c129a3c5cb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:24.975744 30321 consensus_queue.cc:1035] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:24.975621 30367 consensus_queue.cc:237] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:24.978368 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:24.979290 30107 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 1.
I20260806 20:14:24.980407 30150 leader_election.cc:302] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:24.981082 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:24.981190 30321 raft_consensus.cc:2781] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:24.981768 30266 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:24.982338 30321 raft_consensus.cc:686] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Becoming Leader. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:24.983172 30321 consensus_queue.cc:237] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:24.981153 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:24.989028 30266 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 1.
I20260806 20:14:24.992125 30229 leader_election.cc:302] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46; no voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:24.992933 30367 raft_consensus.cc:2726] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:24.993669 29998 catalog_manager.cc:5495] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dad604353364122ae41c129a3c5cb46 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "6dad604353364122ae41c129a3c5cb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:24.995539 29998 catalog_manager.cc:5495] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dad604353364122ae41c129a3c5cb46 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "6dad604353364122ae41c129a3c5cb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:25.000404 30000 catalog_manager.cc:5495] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 975e4cda6abe46838b148df041cc3b2d (127.28.186.194). New cstate: current_term: 1 leader_uuid: "975e4cda6abe46838b148df041cc3b2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:25.009760 30372 consensus_queue.cc:1035] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:25.022267 30339 consensus_queue.cc:1035] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.034397 30339 consensus_queue.cc:1035] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.122695 30372 raft_consensus.cc:482] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:25.123126 30372 raft_consensus.cc:504] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.124625 30372 leader_election.cc:288] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:25.125594 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:25.125618 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:25.126209 30107 raft_consensus.cc:2454] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 0.
I20260806 20:14:25.126273 30266 raft_consensus.cc:2454] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 0.
I20260806 20:14:25.127265 30150 leader_election.cc:302] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:25.127907 30372 raft_consensus.cc:2781] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:25.128156 30372 raft_consensus.cc:482] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:25.128405 30372 raft_consensus.cc:3037] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:25.132906 30372 raft_consensus.cc:504] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.134418 30372 leader_election.cc:288] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:25.135303 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:25.135388 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:25.135784 30107 raft_consensus.cc:3037] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:25.135839 30266 raft_consensus.cc:3037] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:25.140219 30266 raft_consensus.cc:2454] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 1.
I20260806 20:14:25.140365 30107 raft_consensus.cc:2454] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 1.
I20260806 20:14:25.141132 30152 leader_election.cc:302] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:25.141902 30372 raft_consensus.cc:2781] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:25.142269 30372 raft_consensus.cc:686] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Becoming Leader. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:25.142869 30372 consensus_queue.cc:237] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.150264 30000 catalog_manager.cc:5495] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 975e4cda6abe46838b148df041cc3b2d (127.28.186.194). New cstate: current_term: 1 leader_uuid: "975e4cda6abe46838b148df041cc3b2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:25.156137 30339 consensus_queue.cc:1035] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.166551 30339 consensus_queue.cc:1035] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.194398 30339 consensus_queue.cc:1035] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.204308 30357 consensus_queue.cc:1035] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:25.279457 30266 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
mode: GRACEFUL
new_leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:44850
I20260806 20:14:25.280051 30266 raft_consensus.cc:595] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Received request to transfer leadership to TS b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:25.298281 30187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
mode: GRACEFUL
new_leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:38164
I20260806 20:14:25.298804 30187 raft_consensus.cc:595] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Received request to transfer leadership to TS b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:25.303455 30187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "206cc589130d4950abb1a8cf15116498"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
mode: GRACEFUL
new_leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:38164
I20260806 20:14:25.303951 30187 raft_consensus.cc:595] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Received request to transfer leadership to TS b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:25.308455 30357 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.308876 30266 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
mode: GRACEFUL
new_leader_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:44850
I20260806 20:14:25.309409 30266 raft_consensus.cc:595] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Received request to transfer leadership to TS 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:25.319942 30336 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.333575 30107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
mode: GRACEFUL
new_leader_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:33332
I20260806 20:14:25.334100 30107 raft_consensus.cc:595] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Received request to transfer leadership to TS 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:25.338944 30107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e99ec4f4772544638e34273d315381e8"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
mode: GRACEFUL
new_leader_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:33332
I20260806 20:14:25.339545 30107 raft_consensus.cc:595] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Received request to transfer leadership to TS 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:25.422127 30321 raft_consensus.cc:979] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d: : Instructing follower b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:25.422622 30372 raft_consensus.cc:1067] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Signalling peer b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:25.423996 30107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "206cc589130d4950abb1a8cf15116498"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:33302
I20260806 20:14:25.424453 30107 raft_consensus.cc:482] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:25.424719 30107 raft_consensus.cc:3037] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.429229 30107 raft_consensus.cc:504] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.430721 30107 leader_election.cc:288] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:25.431540 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206cc589130d4950abb1a8cf15116498" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:25.431656 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206cc589130d4950abb1a8cf15116498" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:25.432191 30187 raft_consensus.cc:3032] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:25.432238 30266 raft_consensus.cc:3037] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.432487 30187 raft_consensus.cc:729] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:25.433178 30187 consensus_queue.cc:260] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.434105 30187 raft_consensus.cc:3037] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.436784 30266 raft_consensus.cc:2454] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 2.
I20260806 20:14:25.437779 30072 leader_election.cc:302] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:25.438408 30336 raft_consensus.cc:2781] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:25.438612 30187 raft_consensus.cc:2454] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 2.
I20260806 20:14:25.438759 30336 raft_consensus.cc:686] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:25.439604 30336 consensus_queue.cc:237] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.445554 29998 catalog_manager.cc:5495] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 1 to 2, leader changed from 975e4cda6abe46838b148df041cc3b2d (127.28.186.194) to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 2 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:25.454651 30367 consensus_queue.cc:1035] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.460515 30365 raft_consensus.cc:979] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46: : Instructing follower 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:25.460985 30366 raft_consensus.cc:1067] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Signalling peer 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:25.463694 30186 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:38150
I20260806 20:14:25.464286 30186 raft_consensus.cc:482] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:25.464643 30186 raft_consensus.cc:3037] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.465343 30365 consensus_queue.cc:1035] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.469049 30365 consensus_queue.cc:1035] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:25.471340 30186 raft_consensus.cc:504] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.475587 30186 leader_election.cc:288] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:25.476742 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:25.477308 30109 raft_consensus.cc:3037] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.477630 30321 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.481716 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:25.482336 30266 raft_consensus.cc:3032] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:25.482659 30266 raft_consensus.cc:729] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:25.483479 30266 consensus_queue.cc:260] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.484607 30266 raft_consensus.cc:3037] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.489164 30321 raft_consensus.cc:979] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d: : Instructing follower b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:25.489289 30365 consensus_queue.cc:1035] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:25.489833 30109 raft_consensus.cc:2454] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 2.
I20260806 20:14:25.491922 30150 leader_election.cc:302] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:25.492762 30333 raft_consensus.cc:2781] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:25.489639 30321 raft_consensus.cc:1067] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Signalling peer b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:25.497542 30266 raft_consensus.cc:2454] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 2.
I20260806 20:14:25.499445 30107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:33302
I20260806 20:14:25.499984 30107 raft_consensus.cc:482] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:25.500339 30107 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.502211 30365 consensus_queue.cc:1035] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.508735 30333 raft_consensus.cc:686] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Becoming Leader. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:25.509615 30333 consensus_queue.cc:237] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.511976 30107 raft_consensus.cc:504] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.512537 30372 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.514158 30109 raft_consensus.cc:1226] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Rejecting Update request from peer 975e4cda6abe46838b148df041cc3b2d for earlier term 1. Current term is 2. Ops: []
I20260806 20:14:25.515726 30372 consensus_queue.cc:1046] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.516558 30333 raft_consensus.cc:3032] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:25.517557 30333 raft_consensus.cc:729] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:25.518386 30333 consensus_queue.cc:260] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.519362 30333 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.521027 30000 catalog_manager.cc:5495] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d reported cstate change: term changed from 1 to 2, leader changed from 6dad604353364122ae41c129a3c5cb46 (127.28.186.195) to 975e4cda6abe46838b148df041cc3b2d (127.28.186.194). New cstate: current_term: 2 leader_uuid: "975e4cda6abe46838b148df041cc3b2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:25.523151 30107 leader_election.cc:288] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:25.528128 30267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:25.528563 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:25.534276 30333 consensus_queue.cc:1035] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.537684 30187 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 2.
I20260806 20:14:25.538933 30070 leader_election.cc:302] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters: 6dad604353364122ae41c129a3c5cb46
I20260806 20:14:25.540403 30336 raft_consensus.cc:2781] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:25.540457 30365 consensus_queue.cc:1035] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:25.540913 30336 raft_consensus.cc:686] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:25.542073 30336 consensus_queue.cc:237] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.551458 30000 catalog_manager.cc:5495] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 1 to 2, leader changed from 975e4cda6abe46838b148df041cc3b2d (127.28.186.194) to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 2 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:25.555714 30404 raft_consensus.cc:979] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46: : Instructing follower b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:25.556262 30365 raft_consensus.cc:1067] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Signalling peer b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:25.558144 30107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:33316
I20260806 20:14:25.558599 30107 raft_consensus.cc:482] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:25.558892 30107 raft_consensus.cc:3037] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.563510 30333 consensus_queue.cc:1035] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:25.566233 30107 raft_consensus.cc:504] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:25.568276 30107 leader_election.cc:288] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 election: Requested vote from peers 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:25.569047 30267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:25.570137 30267 raft_consensus.cc:3032] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:25.570485 30267 raft_consensus.cc:729] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:25.571298 30267 consensus_queue.cc:260] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:25.572628 30267 raft_consensus.cc:3037] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.573714 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:25.574326 30187 raft_consensus.cc:3037] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.580478 30187 raft_consensus.cc:2454] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 2.
I20260806 20:14:25.580718 30267 raft_consensus.cc:2454] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 2.
I20260806 20:14:25.581614 30070 leader_election.cc:302] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:25.582353 30336 raft_consensus.cc:2781] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:25.582819 30336 raft_consensus.cc:686] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:25.583595 30336 consensus_queue.cc:237] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:25.590909 29998 catalog_manager.cc:5495] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 1 to 2, leader changed from 6dad604353364122ae41c129a3c5cb46 (127.28.186.195) to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 2 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:25.677332 30357 raft_consensus.cc:979] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d: : Instructing follower 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:25.677798 30336 raft_consensus.cc:1067] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Signalling peer 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:25.679231 30187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e99ec4f4772544638e34273d315381e8"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:38148
I20260806 20:14:25.679780 30187 raft_consensus.cc:482] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:25.680140 30187 raft_consensus.cc:3037] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.684713 30187 raft_consensus.cc:504] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.686918 30187 leader_election.cc:288] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:25.687269 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99ec4f4772544638e34273d315381e8" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:25.687875 30107 raft_consensus.cc:3032] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:25.688197 30107 raft_consensus.cc:729] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:25.688328 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99ec4f4772544638e34273d315381e8" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:25.688862 30266 raft_consensus.cc:3037] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.688993 30107 consensus_queue.cc:260] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.690294 30107 raft_consensus.cc:3037] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.695462 30266 raft_consensus.cc:2454] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 2.
I20260806 20:14:25.696488 30107 raft_consensus.cc:2454] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 2.
I20260806 20:14:25.696570 30152 leader_election.cc:302] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:25.697396 30333 raft_consensus.cc:2781] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:25.697883 30333 raft_consensus.cc:686] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Becoming Leader. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:25.698621 30333 consensus_queue.cc:237] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.704424 29998 catalog_manager.cc:5495] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d reported cstate change: term changed from 1 to 2, leader changed from b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193) to 975e4cda6abe46838b148df041cc3b2d (127.28.186.194). New cstate: current_term: 2 leader_uuid: "975e4cda6abe46838b148df041cc3b2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:25.835644 30336 raft_consensus.cc:979] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d: : Instructing follower 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:25.836040 30316 raft_consensus.cc:1067] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Signalling peer 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:25.837381 30187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:38148
I20260806 20:14:25.837884 30187 raft_consensus.cc:482] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:25.838165 30187 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.842818 30187 raft_consensus.cc:504] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.844241 30187 leader_election.cc:288] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:25.845268 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:25.845104 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:25.845835 30266 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.845903 30107 raft_consensus.cc:3032] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:25.846218 30107 raft_consensus.cc:729] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:25.846845 30107 consensus_queue.cc:260] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.847803 30107 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:25.850469 30266 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 2.
I20260806 20:14:25.851409 30152 leader_election.cc:302] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:25.852130 30333 raft_consensus.cc:2781] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:25.852492 30333 raft_consensus.cc:686] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Becoming Leader. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:25.852746 30107 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 2.
I20260806 20:14:25.853303 30333 consensus_queue.cc:237] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:25.859696 29998 catalog_manager.cc:5495] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d reported cstate change: term changed from 1 to 2, leader changed from b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193) to 975e4cda6abe46838b148df041cc3b2d (127.28.186.194). New cstate: current_term: 2 leader_uuid: "975e4cda6abe46838b148df041cc3b2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:25.905335 30107 raft_consensus.cc:1261] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:25.906618 30333 consensus_queue.cc:1035] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:25.918246 30266 raft_consensus.cc:1261] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:25.920011 30333 consensus_queue.cc:1035] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:14:25.959643 30187 raft_consensus.cc:1261] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:25.960948 30316 consensus_queue.cc:1035] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:25.968395 30266 raft_consensus.cc:1261] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:25.971570 30316 consensus_queue.cc:1035] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:26.061896 30266 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:26.067210 30266 raft_consensus.cc:1261] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:26.068521 30316 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:26.077766 30187 raft_consensus.cc:1261] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:26.079039 30316 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:26.119395 30187 raft_consensus.cc:1261] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:26.120589 30316 consensus_queue.cc:1035] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:26.127992 30267 raft_consensus.cc:1261] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:26.132910 30357 consensus_queue.cc:1035] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:26.298220 30107 raft_consensus.cc:1261] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:26.299466 30333 consensus_queue.cc:1035] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:26.307188 30267 raft_consensus.cc:1261] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:26.309377 30333 consensus_queue.cc:1035] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:26.448770 30107 raft_consensus.cc:1261] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:26.450019 30333 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:26.457633 30267 raft_consensus.cc:1261] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:26.459447 30372 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:28.371268 29419 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260806 20:14:28.371946 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:28.396311 30107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
mode: GRACEFUL
new_leader_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:33722
I20260806 20:14:28.396880 30107 raft_consensus.cc:595] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Received request to transfer leadership to TS 6dad604353364122ae41c129a3c5cb46
I20260806 20:14:28.399019 30107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
mode: GRACEFUL
new_leader_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:33722
I20260806 20:14:28.399479 30107 raft_consensus.cc:595] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Received request to transfer leadership to TS 6dad604353364122ae41c129a3c5cb46
I20260806 20:14:28.400321 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 20:14:28.400703 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:28.559626 30419 raft_consensus.cc:979] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d: : Instructing follower 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:28.560122 30416 raft_consensus.cc:1067] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Signalling peer 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:28.561510 30267 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:44842
I20260806 20:14:28.561975 30267 raft_consensus.cc:482] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:28.562252 30267 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:28.566905 30267 raft_consensus.cc:504] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:28.568532 30267 leader_election.cc:288] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 3 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:28.569552 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:28.570180 30107 raft_consensus.cc:3032] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:28.570468 30107 raft_consensus.cc:729] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:28.570724 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:28.571282 30186 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:28.571197 30107 consensus_queue.cc:260] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:28.572463 30107 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:28.576650 30186 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 3.
I20260806 20:14:28.577406 30107 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 3.
I20260806 20:14:28.577736 30229 leader_election.cc:302] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:28.578462 30365 raft_consensus.cc:2781] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:28.579556 30365 raft_consensus.cc:686] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Becoming Leader. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:28.580317 30365 consensus_queue.cc:237] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:28.586794 29998 catalog_manager.cc:5495] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 reported cstate change: term changed from 2 to 3, leader changed from b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193) to 6dad604353364122ae41c129a3c5cb46 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "6dad604353364122ae41c129a3c5cb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:28.594218 30419 raft_consensus.cc:979] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d: : Instructing follower 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:28.595477 30416 raft_consensus.cc:1067] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Signalling peer 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:28.597026 30267 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:44842
I20260806 20:14:28.597590 30267 raft_consensus.cc:482] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:28.597940 30267 raft_consensus.cc:3037] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:28.602281 30267 raft_consensus.cc:504] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:28.603690 30267 leader_election.cc:288] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 2 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:28.604558 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:28.604702 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:28.605232 30186 raft_consensus.cc:3037] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:28.605202 30107 raft_consensus.cc:3032] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:28.605772 30107 raft_consensus.cc:729] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:28.606549 30107 consensus_queue.cc:260] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:28.607769 30107 raft_consensus.cc:3037] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:28.611750 30186 raft_consensus.cc:2454] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 2.
I20260806 20:14:28.612789 30229 leader_election.cc:302] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:28.613427 30365 raft_consensus.cc:2781] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:28.613714 30107 raft_consensus.cc:2454] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 2.
I20260806 20:14:28.613922 30365 raft_consensus.cc:686] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 2 LEADER]: Becoming Leader. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:28.614686 30365 consensus_queue.cc:237] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:28.620836 29998 catalog_manager.cc:5495] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 reported cstate change: term changed from 1 to 2, leader changed from b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193) to 6dad604353364122ae41c129a3c5cb46 (127.28.186.195). New cstate: current_term: 2 leader_uuid: "6dad604353364122ae41c129a3c5cb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:28.970425 30186 raft_consensus.cc:1261] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:28.971864 30365 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:28.982077 30107 raft_consensus.cc:1261] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:28.983716 30365 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:29.100152 30107 raft_consensus.cc:1261] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:29.100314 30186 raft_consensus.cc:1261] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:29.101306 30421 consensus_queue.cc:1035] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:29.102018 30365 consensus_queue.cc:1035] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:14:29.574765 30072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:33715, user_credentials={real_user=slave}} blocked reactor thread for 209972us
I20260806 20:14:30.020325 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:30.128648 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:30.148378 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:30.251302 30442 raft_consensus.cc:482] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dad604353364122ae41c129a3c5cb46)
I20260806 20:14:30.252022 30442 raft_consensus.cc:504] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:30.254667 30442 leader_election.cc:288] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:30.491575 30443 raft_consensus.cc:482] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dad604353364122ae41c129a3c5cb46)
I20260806 20:14:30.492372 30443 raft_consensus.cc:504] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:30.566429 30442 raft_consensus.cc:482] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b68c081f3ce2459b8d81dc29e3d4620d)
I20260806 20:14:30.567126 30442 raft_consensus.cc:504] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:30.569710 30442 leader_election.cc:288] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:30.588434 30445 raft_consensus.cc:482] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975e4cda6abe46838b148df041cc3b2d)
I20260806 20:14:30.589072 30445 raft_consensus.cc:504] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:30.594383 30443 leader_election.cc:288] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:30.593587 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:30.598097 30445 leader_election.cc:288] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:30.602084 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:30.601497 30444 raft_consensus.cc:482] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975e4cda6abe46838b148df041cc3b2d)
I20260806 20:14:30.601766 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:30.605566 30109 raft_consensus.cc:2454] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 1.
I20260806 20:14:30.605253 30444 raft_consensus.cc:504] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:30.602743 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99ec4f4772544638e34273d315381e8" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:30.600817 30446 raft_consensus.cc:482] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975e4cda6abe46838b148df041cc3b2d)
I20260806 20:14:30.610273 30446 raft_consensus.cc:504] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:30.613509 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:30.615037 30444 leader_election.cc:288] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:30.617009 30150 leader_election.cc:302] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:30.617552 30446 leader_election.cc:288] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:30.617939 30442 raft_consensus.cc:2781] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:14:30.618362 30442 raft_consensus.cc:482] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6dad604353364122ae41c129a3c5cb46)
I20260806 20:14:30.618876 30442 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:30.619541 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:30.625751 30442 raft_consensus.cc:504] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:30.628633 30108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:30.629634 30442 leader_election.cc:288] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:30.630975 30449 raft_consensus.cc:482] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 975e4cda6abe46838b148df041cc3b2d)
I20260806 20:14:30.632390 30449 raft_consensus.cc:504] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:30.634862 30449 leader_election.cc:288] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:30.638578 30267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:30.640587 30264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e99ec4f4772544638e34273d315381e8" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:30.648533 30263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:30.649923 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:30.651511 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:30.653559 30108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:30.654314 30266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:30.654754 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:30.656663 30152 leader_election.cc:302] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d; no voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:30.656925 30072 leader_election.cc:302] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b68c081f3ce2459b8d81dc29e3d4620d; no voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:30.658344 30072 leader_election.cc:302] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b68c081f3ce2459b8d81dc29e3d4620d; no voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:30.659530 30262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:30.665565 30072 leader_election.cc:302] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b68c081f3ce2459b8d81dc29e3d4620d; no voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:30.670650 30072 leader_election.cc:302] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b68c081f3ce2459b8d81dc29e3d4620d; no voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:30.671909 30229 leader_election.cc:302] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46; no voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:30.673707 30152 leader_election.cc:302] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d; no voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:30.673656 30441 raft_consensus.cc:2726] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:14:30.675429 30454 raft_consensus.cc:2726] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:14:30.675824 30439 raft_consensus.cc:2726] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:14:30.666167 30416 raft_consensus.cc:2726] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:14:30.676671 30447 raft_consensus.cc:2726] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:14:30.676684 30416 raft_consensus.cc:2726] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:14:30.665752 30440 raft_consensus.cc:2726] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:14:30.668217 30452 raft_consensus.cc:482] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 975e4cda6abe46838b148df041cc3b2d)
I20260806 20:14:30.682016 30452 raft_consensus.cc:504] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:30.683791 30452 leader_election.cc:288] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:30.684590 30109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:30.684760 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:30.686106 30229 leader_election.cc:302] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46; no voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:30.686795 30452 raft_consensus.cc:2726] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:14:30.887172 30212 debug-util.cc:398] Leaking SignalData structure 0x7b08001185c0 after lost signal to thread 29422
W20260806 20:14:30.887925 30212 debug-util.cc:398] Leaking SignalData structure 0x7b0800118ca0 after lost signal to thread 30036
W20260806 20:14:30.888677 30212 debug-util.cc:398] Leaking SignalData structure 0x7b08001a02c0 after lost signal to thread 30134
W20260806 20:14:30.889354 30212 debug-util.cc:398] Leaking SignalData structure 0x7b08000cca60 after lost signal to thread 30215
W20260806 20:14:30.890182 30212 debug-util.cc:398] Leaking SignalData structure 0x7b080009f160 after lost signal to thread 30294
I20260806 20:14:31.193830 30187 raft_consensus.cc:1226] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Rejecting Update request from peer 6dad604353364122ae41c129a3c5cb46 for earlier term 1. Current term is 2. Ops: []
I20260806 20:14:31.195186 30452 consensus_queue.cc:1046] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:31.196723 30455 raft_consensus.cc:3032] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:31.197098 30455 raft_consensus.cc:729] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:31.197937 30455 consensus_queue.cc:260] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:31.199115 30455 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:31.240245 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:31.245607 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:31.246055 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:32.246932 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:32.252275 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:32.252687 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:32.382136 30454 raft_consensus.cc:482] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:32.383268 30454 raft_consensus.cc:504] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:32.387727 30454 leader_election.cc:288] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:32.388914 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" is_pre_election: true
I20260806 20:14:32.389773 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:32.390908 30265 raft_consensus.cc:2454] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 2.
I20260806 20:14:32.392581 30152 leader_election.cc:302] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters: b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:32.394119 30454 raft_consensus.cc:2781] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:14:32.394574 30454 raft_consensus.cc:482] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:32.395329 30454 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:32.404598 30454 raft_consensus.cc:504] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:32.406973 30454 leader_election.cc:288] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 3 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:32.407545 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:32.407699 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:32.408346 30265 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:32.416066 30265 raft_consensus.cc:2454] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 3.
I20260806 20:14:32.417155 30152 leader_election.cc:302] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters: b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:32.417924 30454 raft_consensus.cc:2781] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:32.419023 30454 raft_consensus.cc:686] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 3 LEADER]: Becoming Leader. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:32.420259 30454 consensus_queue.cc:237] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:32.434684 29999 catalog_manager.cc:5495] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d reported cstate change: term changed from 1 to 3, leader changed from 6dad604353364122ae41c129a3c5cb46 (127.28.186.195) to 975e4cda6abe46838b148df041cc3b2d (127.28.186.194). New cstate: current_term: 3 leader_uuid: "975e4cda6abe46838b148df041cc3b2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:32.562417 30457 raft_consensus.cc:482] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6dad604353364122ae41c129a3c5cb46)
I20260806 20:14:32.562985 30457 raft_consensus.cc:504] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:32.564905 30457 leader_election.cc:288] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:32.565961 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "975e4cda6abe46838b148df041cc3b2d" is_pre_election: true
I20260806 20:14:32.566121 30265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6dad604353364122ae41c129a3c5cb46" is_pre_election: true
I20260806 20:14:32.566991 30265 raft_consensus.cc:2354] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b68c081f3ce2459b8d81dc29e3d4620d for earlier term 2. Current term is 3.
I20260806 20:14:32.568120 30072 leader_election.cc:398] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715) with higher term. Message: Invalid argument: T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b68c081f3ce2459b8d81dc29e3d4620d for earlier term 2. Current term is 3.
I20260806 20:14:32.568357 30072 leader_election.cc:401] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:14:32.568925 30457 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:14:32.577068 30457 raft_consensus.cc:2726] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715) with higher term. Message: Invalid argument: T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b68c081f3ce2459b8d81dc29e3d4620d for earlier term 2. Current term is 3.
I20260806 20:14:32.917727 30265 raft_consensus.cc:1261] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:14:32.918983 30454 consensus_queue.cc:1035] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:32.933985 30107 raft_consensus.cc:1261] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:14:32.937711 30454 consensus_queue.cc:1035] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:14:33.253747 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:33.272658 30186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
mode: GRACEFUL
new_leader_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:51268
I20260806 20:14:33.273340 30186 raft_consensus.cc:595] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Received request to transfer leadership to TS 6dad604353364122ae41c129a3c5cb46
I20260806 20:14:33.274674 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260806 20:14:33.275111 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:33.775138 30454 raft_consensus.cc:979] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d: : Instructing follower 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:33.775672 30480 raft_consensus.cc:1067] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Signalling peer 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:33.777065 30265 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:44828
I20260806 20:14:33.777575 30265 raft_consensus.cc:482] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:33.777829 30265 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:33.782564 30265 raft_consensus.cc:504] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:33.784088 30265 leader_election.cc:288] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 3 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:33.785028 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:33.785262 30186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:33.785517 30107 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:33.785811 30186 raft_consensus.cc:3032] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:33.786057 30186 raft_consensus.cc:729] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:33.786747 30186 consensus_queue.cc:260] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:33.787760 30186 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:33.790199 30107 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 3.
I20260806 20:14:33.791265 30229 leader_election.cc:302] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:33.792122 30455 raft_consensus.cc:2781] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:33.792903 30186 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 3.
I20260806 20:14:33.793107 30455 raft_consensus.cc:686] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Becoming Leader. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:33.794001 30455 consensus_queue.cc:237] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:33.800859 29999 catalog_manager.cc:5495] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 reported cstate change: term changed from 2 to 3, leader changed from 975e4cda6abe46838b148df041cc3b2d (127.28.186.194) to 6dad604353364122ae41c129a3c5cb46 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "6dad604353364122ae41c129a3c5cb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:34.274706 30107 raft_consensus.cc:1261] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:34.275424 30186 raft_consensus.cc:1261] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:34.276134 30455 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:14:34.276662 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:34.277076 30481 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:34.285574 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:34.286319 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:35.287263 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:35.292603 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:35.293033 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:36.293810 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:36.299449 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:36.299858 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:37.300727 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:37.306023 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:37.306458 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:38.307307 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:38.312517 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:38.312932 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:39.342777 29419 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260806 20:14:39.349573 30262 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
mode: GRACEFUL
new_leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:44850
I20260806 20:14:39.350095 30262 raft_consensus.cc:595] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 2 LEADER]: Received request to transfer leadership to TS b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:39.357359 30262 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
mode: GRACEFUL
new_leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:44850
I20260806 20:14:39.357926 30262 raft_consensus.cc:595] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Received request to transfer leadership to TS b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:39.365108 30187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
mode: GRACEFUL
new_leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:38164
I20260806 20:14:39.365607 30187 raft_consensus.cc:595] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Received request to transfer leadership to TS b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:39.369776 30187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
mode: GRACEFUL
new_leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:38164
I20260806 20:14:39.370187 30187 raft_consensus.cc:595] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Received request to transfer leadership to TS b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:39.374291 30262 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
mode: GRACEFUL
new_leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:44850
I20260806 20:14:39.374794 30262 raft_consensus.cc:595] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Received request to transfer leadership to TS b68c081f3ce2459b8d81dc29e3d4620d
I20260806 20:14:39.380882 30187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
mode: GRACEFUL
new_leader_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:38164
I20260806 20:14:39.381317 30187 raft_consensus.cc:595] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 3 LEADER]: Received request to transfer leadership to TS 6dad604353364122ae41c129a3c5cb46
I20260806 20:14:39.433408 30481 raft_consensus.cc:979] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46: : Instructing follower b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.433835 30497 raft_consensus.cc:1067] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Signalling peer b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.435303 30107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:33316
I20260806 20:14:39.435770 30107 raft_consensus.cc:482] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:39.436017 30107 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:39.440533 30107 raft_consensus.cc:504] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.442358 30107 leader_election.cc:288] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 4 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:39.443190 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:39.443434 30262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:39.443691 30187 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:39.444007 30262 raft_consensus.cc:3032] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:14:39.444263 30262 raft_consensus.cc:729] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:39.444868 30262 consensus_queue.cc:260] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.445839 30262 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:39.448796 30187 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 4.
I20260806 20:14:39.449949 30070 leader_election.cc:302] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:39.450701 30495 raft_consensus.cc:2781] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:14:39.451601 30262 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 4.
I20260806 20:14:39.451740 30495 raft_consensus.cc:686] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:39.452709 30495 consensus_queue.cc:237] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.458838 29999 catalog_manager.cc:5495] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 3 to 4, leader changed from 6dad604353364122ae41c129a3c5cb46 (127.28.186.195) to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 4 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:39.471370 30481 raft_consensus.cc:979] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46: : Instructing follower b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.471871 30497 raft_consensus.cc:1067] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Signalling peer b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.473145 30107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:33316
I20260806 20:14:39.473580 30107 raft_consensus.cc:482] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:39.473819 30107 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:39.478129 30107 raft_consensus.cc:504] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.479457 30107 leader_election.cc:288] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 4 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:39.480279 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:39.480432 30262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:39.480747 30187 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:39.480942 30262 raft_consensus.cc:3032] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:14:39.481225 30262 raft_consensus.cc:729] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:39.481887 30262 consensus_queue.cc:260] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.482834 30262 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:39.485008 30187 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 4.
I20260806 20:14:39.485913 30070 leader_election.cc:302] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:39.486662 30495 raft_consensus.cc:2781] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:14:39.487020 30495 raft_consensus.cc:686] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:39.487910 30262 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 4.
I20260806 20:14:39.487824 30495 consensus_queue.cc:237] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.494009 30506 raft_consensus.cc:979] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d: : Instructing follower b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.494473 30480 raft_consensus.cc:1067] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Signalling peer b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.495224 29999 catalog_manager.cc:5495] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 3 to 4, leader changed from 6dad604353364122ae41c129a3c5cb46 (127.28.186.195) to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 4 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:39.495689 30107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:33302
I20260806 20:14:39.496167 30107 raft_consensus.cc:482] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:39.496409 30107 raft_consensus.cc:3037] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:39.501684 30107 raft_consensus.cc:504] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.503355 30107 leader_election.cc:288] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 3 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:39.503962 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:39.504396 30187 raft_consensus.cc:3032] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:39.504596 30187 raft_consensus.cc:729] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:39.505144 30187 consensus_queue.cc:260] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.506079 30187 raft_consensus.cc:3037] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:39.507107 30262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d187995bdd1494c89d80604e6f65944" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:39.507673 30262 raft_consensus.cc:3037] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:39.512111 30187 raft_consensus.cc:2454] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 3.
I20260806 20:14:39.513060 30070 leader_election.cc:302] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:39.513715 30495 raft_consensus.cc:2781] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:39.513685 30262 raft_consensus.cc:2454] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 3.
I20260806 20:14:39.514321 30495 raft_consensus.cc:686] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:39.515130 30495 consensus_queue.cc:237] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.521692 29999 catalog_manager.cc:5495] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 2 to 3, leader changed from 975e4cda6abe46838b148df041cc3b2d (127.28.186.194) to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 3 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:39.687494 30481 raft_consensus.cc:979] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46: : Instructing follower b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.687927 30497 raft_consensus.cc:1067] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 2 LEADER]: Signalling peer b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.689231 30107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:33316
I20260806 20:14:39.689705 30107 raft_consensus.cc:482] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:39.689981 30107 raft_consensus.cc:3037] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:39.695243 30107 raft_consensus.cc:504] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.697041 30107 leader_election.cc:288] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 3 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:39.697693 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:39.698190 30187 raft_consensus.cc:3037] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:39.698032 30262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48f8dee5c9d48498fc4a60b781686c2" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:39.698606 30262 raft_consensus.cc:3032] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:39.698844 30262 raft_consensus.cc:729] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:39.699512 30262 consensus_queue.cc:260] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.700651 30262 raft_consensus.cc:3037] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:39.702637 30187 raft_consensus.cc:2454] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 3.
I20260806 20:14:39.703433 30070 leader_election.cc:302] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 975e4cda6abe46838b148df041cc3b2d, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:39.704030 30495 raft_consensus.cc:2781] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:39.704488 30495 raft_consensus.cc:686] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:39.705371 30495 consensus_queue.cc:237] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.705785 30262 raft_consensus.cc:2454] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 3.
I20260806 20:14:39.712412 29999 catalog_manager.cc:5495] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 2 to 3, leader changed from 6dad604353364122ae41c129a3c5cb46 (127.28.186.195) to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 3 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:39.728981 30506 raft_consensus.cc:979] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d: : Instructing follower b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.729416 30480 raft_consensus.cc:1067] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Signalling peer b68c081f3ce2459b8d81dc29e3d4620d to start an election
I20260806 20:14:39.730712 30107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
from {username='slave'} at 127.0.0.1:33302
I20260806 20:14:39.731149 30107 raft_consensus.cc:482] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:39.731391 30107 raft_consensus.cc:3037] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:39.735677 30107 raft_consensus.cc:504] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.737092 30107 leader_election.cc:288] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 election: Requested vote from peers 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:39.737888 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:39.738142 30262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:39.738466 30187 raft_consensus.cc:3032] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:14:39.738648 30262 raft_consensus.cc:3037] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:39.738822 30187 raft_consensus.cc:729] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:39.739621 30187 consensus_queue.cc:260] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.740856 30187 raft_consensus.cc:3037] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:39.745587 30262 raft_consensus.cc:2454] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 2.
I20260806 20:14:39.746764 30072 leader_election.cc:302] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:39.747171 30187 raft_consensus.cc:2454] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68c081f3ce2459b8d81dc29e3d4620d in term 2.
I20260806 20:14:39.747619 30495 raft_consensus.cc:2781] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:39.748070 30495 raft_consensus.cc:686] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Becoming Leader. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:39.748850 30495 consensus_queue.cc:237] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.755668 29999 catalog_manager.cc:5495] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d reported cstate change: term changed from 1 to 2, leader changed from 975e4cda6abe46838b148df041cc3b2d (127.28.186.194) to b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193). New cstate: current_term: 2 leader_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:39.823220 30506 raft_consensus.cc:979] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d: : Instructing follower 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:39.823619 30480 raft_consensus.cc:1067] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 3 LEADER]: Signalling peer 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:39.824901 30262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:44828
I20260806 20:14:39.825403 30262 raft_consensus.cc:482] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:39.825711 30262 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:39.830711 30262 raft_consensus.cc:504] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.832530 30262 leader_election.cc:288] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 4 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:39.833510 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:39.833604 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef940c78b6f4978a1802a7fffe371db" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:39.834162 30187 raft_consensus.cc:3032] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:14:39.834197 30107 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:39.834553 30187 raft_consensus.cc:729] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:39.835168 30187 consensus_queue.cc:260] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.836171 30187 raft_consensus.cc:3037] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:14:39.839524 30107 raft_consensus.cc:2454] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 4.
I20260806 20:14:39.840370 30229 leader_election.cc:302] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, b68c081f3ce2459b8d81dc29e3d4620d; no voters:
I20260806 20:14:39.840950 30497 raft_consensus.cc:2781] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:14:39.841197 30187 raft_consensus.cc:2454] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 4.
I20260806 20:14:39.841307 30497 raft_consensus.cc:686] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 4 LEADER]: Becoming Leader. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:39.842183 30497 consensus_queue.cc:237] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:39.849074 29999 catalog_manager.cc:5495] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 reported cstate change: term changed from 3 to 4, leader changed from 975e4cda6abe46838b148df041cc3b2d (127.28.186.194) to 6dad604353364122ae41c129a3c5cb46 (127.28.186.195). New cstate: current_term: 4 leader_uuid: "6dad604353364122ae41c129a3c5cb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:39.910946 30262 raft_consensus.cc:1261] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:14:39.912132 30495 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:14:39.921139 30262 raft_consensus.cc:1261] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:39.923022 30187 raft_consensus.cc:1261] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:14:39.924530 30498 consensus_queue.cc:1035] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:14:39.924705 30495 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:14:39.945407 30187 raft_consensus.cc:1261] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:39.947242 30514 consensus_queue.cc:1035] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:14:39.972267 30187 raft_consensus.cc:1261] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:14:39.973526 30498 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:14:39.982340 30265 raft_consensus.cc:1261] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:14:39.984088 30498 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:14:40.162617 30265 raft_consensus.cc:1261] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:40.164155 30495 consensus_queue.cc:1035] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:40.166712 30262 raft_consensus.cc:1261] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:40.167930 30498 consensus_queue.cc:1035] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:40.174461 30186 raft_consensus.cc:1261] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:40.174543 30187 raft_consensus.cc:1261] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Refusing update from remote peer b68c081f3ce2459b8d81dc29e3d4620d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:14:40.176182 30514 consensus_queue.cc:1035] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:40.177685 30514 consensus_queue.cc:1035] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:14:40.424913 30107 raft_consensus.cc:1261] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:14:40.425981 30497 consensus_queue.cc:1035] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:40.434566 30187 raft_consensus.cc:1261] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:14:40.435945 30497 consensus_queue.cc:1035] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:42.413614 29419 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260806 20:14:42.414387 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:42.421586 30107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
mode: GRACEFUL
new_leader_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:33722
I20260806 20:14:42.422322 30107 raft_consensus.cc:595] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Received request to transfer leadership to TS 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:42.424741 30107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
mode: GRACEFUL
new_leader_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:33722
I20260806 20:14:42.425210 30107 raft_consensus.cc:595] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Received request to transfer leadership to TS 6dad604353364122ae41c129a3c5cb46
I20260806 20:14:42.427227 30107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
mode: GRACEFUL
new_leader_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:33722
I20260806 20:14:42.427749 30107 raft_consensus.cc:595] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Received request to transfer leadership to TS 975e4cda6abe46838b148df041cc3b2d
I20260806 20:14:42.429831 30107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569"
dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
mode: GRACEFUL
new_leader_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:33722
I20260806 20:14:42.430289 30107 raft_consensus.cc:595] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Received request to transfer leadership to TS 6dad604353364122ae41c129a3c5cb46
I20260806 20:14:42.431216 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 20:14:42.431617 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:42.606938 30498 raft_consensus.cc:979] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d: : Instructing follower 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:42.607455 30535 raft_consensus.cc:1067] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Signalling peer 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:42.608891 30187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:38148
I20260806 20:14:42.609319 30187 raft_consensus.cc:482] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:42.609632 30187 raft_consensus.cc:3037] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:42.615476 30187 raft_consensus.cc:504] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:42.617465 30187 leader_election.cc:288] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 3 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:42.618350 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:42.618569 30262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898d35ff17224b77a542377b8b3fa397" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:42.619081 30107 raft_consensus.cc:3032] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:42.619167 30262 raft_consensus.cc:3037] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:42.619361 30107 raft_consensus.cc:729] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:42.620018 30107 consensus_queue.cc:260] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:42.621019 30107 raft_consensus.cc:3037] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:42.624400 30262 raft_consensus.cc:2454] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 3.
I20260806 20:14:42.625542 30152 leader_election.cc:302] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:42.626127 30107 raft_consensus.cc:2454] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 3.
I20260806 20:14:42.626479 30480 raft_consensus.cc:2781] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:42.627425 30480 raft_consensus.cc:686] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 3 LEADER]: Becoming Leader. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:42.628361 30480 consensus_queue.cc:237] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:42.635010 30000 catalog_manager.cc:5495] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d reported cstate change: term changed from 2 to 3, leader changed from b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193) to 975e4cda6abe46838b148df041cc3b2d (127.28.186.194). New cstate: current_term: 3 leader_uuid: "975e4cda6abe46838b148df041cc3b2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:42.646781 30498 raft_consensus.cc:979] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d: : Instructing follower 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:42.647322 30535 raft_consensus.cc:1067] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Signalling peer 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:42.648693 30262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:44842
I20260806 20:14:42.649092 30262 raft_consensus.cc:482] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:42.649303 30262 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:14:42.654062 30262 raft_consensus.cc:504] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:42.655606 30262 leader_election.cc:288] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 5 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:42.656492 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:42.656749 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89914b5b49b143a98cfe92a893ef4569" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:42.657068 30107 raft_consensus.cc:3032] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:14:42.657305 30187 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:14:42.657351 30107 raft_consensus.cc:729] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:42.658188 30107 consensus_queue.cc:260] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:42.659106 30107 raft_consensus.cc:3037] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:14:42.661784 30187 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 5.
I20260806 20:14:42.662851 30229 leader_election.cc:302] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:42.663715 30530 raft_consensus.cc:2781] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:14:42.663822 30107 raft_consensus.cc:2454] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 5.
I20260806 20:14:42.664974 30530 raft_consensus.cc:686] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 5 LEADER]: Becoming Leader. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:42.665897 30530 consensus_queue.cc:237] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:42.670496 30498 raft_consensus.cc:979] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d: : Instructing follower 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:42.671041 30535 raft_consensus.cc:1067] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Signalling peer 6dad604353364122ae41c129a3c5cb46 to start an election
I20260806 20:14:42.672402 30262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e"
dest_uuid: "6dad604353364122ae41c129a3c5cb46"
from {username='slave'} at 127.0.0.1:44842
I20260806 20:14:42.672921 30262 raft_consensus.cc:482] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:42.673259 30262 raft_consensus.cc:3037] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:42.674192 30000 catalog_manager.cc:5495] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 reported cstate change: term changed from 4 to 5, leader changed from b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193) to 6dad604353364122ae41c129a3c5cb46 (127.28.186.195). New cstate: current_term: 5 leader_uuid: "6dad604353364122ae41c129a3c5cb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:42.678874 30262 raft_consensus.cc:504] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:42.680429 30262 leader_election.cc:288] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 3 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663)
I20260806 20:14:42.681638 30187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
I20260806 20:14:42.682176 30187 raft_consensus.cc:3037] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:42.682176 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aec719ee2834a7ba6c7d822c788ad3e" candidate_uuid: "6dad604353364122ae41c129a3c5cb46" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:42.682838 30107 raft_consensus.cc:3032] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:14:42.683163 30107 raft_consensus.cc:729] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:42.683866 30107 consensus_queue.cc:260] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:42.685050 30107 raft_consensus.cc:3037] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:14:42.688810 30187 raft_consensus.cc:2454] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 3.
I20260806 20:14:42.689895 30229 leader_election.cc:302] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:42.690734 30530 raft_consensus.cc:2781] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:14:42.691046 30107 raft_consensus.cc:2454] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dad604353364122ae41c129a3c5cb46 in term 3.
I20260806 20:14:42.691128 30530 raft_consensus.cc:686] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Becoming Leader. State: Replica: 6dad604353364122ae41c129a3c5cb46, State: Running, Role: LEADER
I20260806 20:14:42.692206 30530 consensus_queue.cc:237] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } }
I20260806 20:14:42.698537 30000 catalog_manager.cc:5495] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 reported cstate change: term changed from 2 to 3, leader changed from b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193) to 6dad604353364122ae41c129a3c5cb46 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "6dad604353364122ae41c129a3c5cb46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:42.760339 30547 raft_consensus.cc:979] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d: : Instructing follower 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:42.762347 30535 raft_consensus.cc:1067] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Signalling peer 975e4cda6abe46838b148df041cc3b2d to start an election
I20260806 20:14:42.763991 30187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66"
dest_uuid: "975e4cda6abe46838b148df041cc3b2d"
from {username='slave'} at 127.0.0.1:38148
I20260806 20:14:42.764565 30187 raft_consensus.cc:482] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:14:42.764909 30187 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:14:42.769552 30187 raft_consensus.cc:504] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:42.772217 30187 leader_election.cc:288] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 5 election: Requested vote from peers b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001), 6dad604353364122ae41c129a3c5cb46 (127.28.186.195:33715)
I20260806 20:14:42.772424 30262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "6dad604353364122ae41c129a3c5cb46"
I20260806 20:14:42.773018 30262 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:14:42.773411 30107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a4033e9b914dd4a00cbfeb3f6b7e66" candidate_uuid: "975e4cda6abe46838b148df041cc3b2d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b68c081f3ce2459b8d81dc29e3d4620d"
I20260806 20:14:42.774019 30107 raft_consensus.cc:3032] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:14:42.774348 30107 raft_consensus.cc:729] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b68c081f3ce2459b8d81dc29e3d4620d, State: Running, Role: LEADER
I20260806 20:14:42.775161 30107 consensus_queue.cc:260] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:42.776487 30107 raft_consensus.cc:3037] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:14:42.779418 30262 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 5.
I20260806 20:14:42.780412 30152 leader_election.cc:302] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dad604353364122ae41c129a3c5cb46, 975e4cda6abe46838b148df041cc3b2d; no voters:
I20260806 20:14:42.781176 30480 raft_consensus.cc:2781] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:14:42.781646 30480 raft_consensus.cc:686] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 5 LEADER]: Becoming Leader. State: Replica: 975e4cda6abe46838b148df041cc3b2d, State: Running, Role: LEADER
I20260806 20:14:42.782488 30480 consensus_queue.cc:237] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } }
I20260806 20:14:42.782979 30107 raft_consensus.cc:2454] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 975e4cda6abe46838b148df041cc3b2d in term 5.
I20260806 20:14:42.789553 30000 catalog_manager.cc:5495] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d reported cstate change: term changed from 4 to 5, leader changed from b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193) to 975e4cda6abe46838b148df041cc3b2d (127.28.186.194). New cstate: current_term: 5 leader_uuid: "975e4cda6abe46838b148df041cc3b2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:43.008257 30262 raft_consensus.cc:1261] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:43.009361 30480 consensus_queue.cc:1035] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:43.018646 30107 raft_consensus.cc:1261] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:43.021025 30552 consensus_queue.cc:1035] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:43.050357 30107 raft_consensus.cc:1261] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 5 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:14:43.051458 30530 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:14:43.061343 30187 raft_consensus.cc:1261] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 5 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:14:43.062610 30557 consensus_queue.cc:1035] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:14:43.078892 30107 raft_consensus.cc:1261] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:43.079841 30557 consensus_queue.cc:1035] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:43.087791 30187 raft_consensus.cc:1261] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Refusing update from remote peer 6dad604353364122ae41c129a3c5cb46: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:14:43.090561 30557 consensus_queue.cc:1035] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "975e4cda6abe46838b148df041cc3b2d" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 36663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:14:43.161628 30107 raft_consensus.cc:1261] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 5 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:14:43.162818 30552 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b68c081f3ce2459b8d81dc29e3d4620d" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:14:43.170959 30262 raft_consensus.cc:1261] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 5 FOLLOWER]: Refusing update from remote peer 975e4cda6abe46838b148df041cc3b2d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 20:14:43.173053 30536 consensus_queue.cc:1035] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dad604353364122ae41c129a3c5cb46" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:14:43.432432 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:43.437556 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:43.437955 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:44.438807 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:44.444224 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:44.444640 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:45.445504 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:45.450623 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:45.451046 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:46.451886 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:46.457553 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:46.458016 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:47.458907 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:47.464608 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:47.465075 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:48.465934 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:48.471323 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:48.471717 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:49.472451 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:49.477643 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:49.478045 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:50.478816 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:50.483935 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:50.484308 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:51.485143 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:14:51.490619 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 20:14:51.491005 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:14:52.523749 29419 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260806 20:14:52.524385 29419 tablet_server.cc:178] TabletServer@127.28.186.193:0 shutting down...
I20260806 20:14:52.558812 29419 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:14:52.559584 29419 tablet_replica.cc:330] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.560523 29419 raft_consensus.cc:2229] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.561436 29419 raft_consensus.cc:2258] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P b68c081f3ce2459b8d81dc29e3d4620d [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.563668 29419 tablet_replica.cc:330] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.564312 29419 raft_consensus.cc:2229] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.564822 29419 raft_consensus.cc:2258] T 9ef940c78b6f4978a1802a7fffe371db P b68c081f3ce2459b8d81dc29e3d4620d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.566726 29419 tablet_replica.cc:330] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.567221 29419 raft_consensus.cc:2229] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.568338 29419 raft_consensus.cc:2258] T e48f8dee5c9d48498fc4a60b781686c2 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.570534 29419 tablet_replica.cc:330] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.571022 29419 raft_consensus.cc:2229] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.571493 29419 raft_consensus.cc:2258] T e99ec4f4772544638e34273d315381e8 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.573302 29419 tablet_replica.cc:330] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.573786 29419 raft_consensus.cc:2229] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.574748 29419 raft_consensus.cc:2258] T 3b6a183f6d714d0ba65ca608304b96d0 P b68c081f3ce2459b8d81dc29e3d4620d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.576593 29419 tablet_replica.cc:330] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.577056 29419 raft_consensus.cc:2229] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.577417 29419 raft_consensus.cc:2258] T 89914b5b49b143a98cfe92a893ef4569 P b68c081f3ce2459b8d81dc29e3d4620d [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.579259 29419 tablet_replica.cc:330] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.579715 29419 raft_consensus.cc:2229] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.580122 29419 raft_consensus.cc:2258] T 898d35ff17224b77a542377b8b3fa397 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.581841 29419 tablet_replica.cc:330] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.582316 29419 raft_consensus.cc:2229] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.583356 29419 raft_consensus.cc:2258] T 4d187995bdd1494c89d80604e6f65944 P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.585072 29419 tablet_replica.cc:330] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.585564 29419 raft_consensus.cc:2229] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.586519 29419 raft_consensus.cc:2258] T 206cc589130d4950abb1a8cf15116498 P b68c081f3ce2459b8d81dc29e3d4620d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.588294 29419 tablet_replica.cc:330] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d: stopping tablet replica
I20260806 20:14:52.588711 29419 raft_consensus.cc:2229] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.589114 29419 raft_consensus.cc:2258] T 7aec719ee2834a7ba6c7d822c788ad3e P b68c081f3ce2459b8d81dc29e3d4620d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.623492 29419 tablet_server.cc:194] TabletServer@127.28.186.193:0 shutdown complete.
I20260806 20:14:52.643899 29419 tablet_server.cc:178] TabletServer@127.28.186.194:0 shutting down...
I20260806 20:14:52.671797 29419 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:14:52.672580 29419 tablet_replica.cc:330] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.673183 29419 raft_consensus.cc:2229] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.674250 29419 raft_consensus.cc:2258] T 898d35ff17224b77a542377b8b3fa397 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.676203 29419 tablet_replica.cc:330] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.676780 29419 raft_consensus.cc:2229] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.677273 29419 raft_consensus.cc:2258] T 9ef940c78b6f4978a1802a7fffe371db P 975e4cda6abe46838b148df041cc3b2d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.680017 29419 tablet_replica.cc:330] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.680748 29419 raft_consensus.cc:2229] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 5 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.681665 29419 raft_consensus.cc:2258] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 975e4cda6abe46838b148df041cc3b2d [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.683662 29419 tablet_replica.cc:330] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.684137 29419 raft_consensus.cc:2229] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.684605 29419 raft_consensus.cc:2258] T 4d187995bdd1494c89d80604e6f65944 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.686450 29419 tablet_replica.cc:330] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.686959 29419 raft_consensus.cc:2229] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.687387 29419 raft_consensus.cc:2258] T 206cc589130d4950abb1a8cf15116498 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.689138 29419 tablet_replica.cc:330] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.689682 29419 raft_consensus.cc:2229] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.690111 29419 raft_consensus.cc:2258] T 7aec719ee2834a7ba6c7d822c788ad3e P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.691751 29419 tablet_replica.cc:330] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.692217 29419 raft_consensus.cc:2229] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.692727 29419 raft_consensus.cc:2258] T e48f8dee5c9d48498fc4a60b781686c2 P 975e4cda6abe46838b148df041cc3b2d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.694411 29419 tablet_replica.cc:330] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.694859 29419 raft_consensus.cc:2229] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 5 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.695403 29419 raft_consensus.cc:2258] T 89914b5b49b143a98cfe92a893ef4569 P 975e4cda6abe46838b148df041cc3b2d [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.697233 29419 tablet_replica.cc:330] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.697710 29419 raft_consensus.cc:2229] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 2 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.698640 29419 raft_consensus.cc:2258] T e99ec4f4772544638e34273d315381e8 P 975e4cda6abe46838b148df041cc3b2d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.700516 29419 tablet_replica.cc:330] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d: stopping tablet replica
I20260806 20:14:52.700960 29419 raft_consensus.cc:2229] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.701362 29419 raft_consensus.cc:2258] T 3b6a183f6d714d0ba65ca608304b96d0 P 975e4cda6abe46838b148df041cc3b2d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 20:14:52.727991 30229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.186.194:36663: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20260806 20:14:52.731180 30229 consensus_peers.cc:487] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 -> Peer 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663): Couldn't send request to peer 975e4cda6abe46838b148df041cc3b2d. Status: Network error: Client connection negotiation failed: client connection to 127.28.186.194:36663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:52.750981 29419 tablet_server.cc:194] TabletServer@127.28.186.194:0 shutdown complete.
I20260806 20:14:52.769982 29419 tablet_server.cc:178] TabletServer@127.28.186.195:0 shutting down...
W20260806 20:14:52.781167 30229 consensus_peers.cc:487] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 -> Peer 975e4cda6abe46838b148df041cc3b2d (127.28.186.194:36663): Couldn't send request to peer 975e4cda6abe46838b148df041cc3b2d. Status: Network error: Client connection negotiation failed: client connection to 127.28.186.194:36663: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:52.803424 29419 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 20:14:52.804425 29419 tablet_replica.cc:330] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.805424 29419 raft_consensus.cc:2229] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.806245 29419 raft_consensus.cc:2258] T 898d35ff17224b77a542377b8b3fa397 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.811812 29419 tablet_replica.cc:330] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.812753 29419 raft_consensus.cc:2229] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.813480 29419 raft_consensus.cc:2258] T e48f8dee5c9d48498fc4a60b781686c2 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.816334 29419 tablet_replica.cc:330] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.817144 29419 raft_consensus.cc:2229] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 5 FOLLOWER]: Raft consensus shutting down.
W20260806 20:14:52.817332 30229 consensus_peers.cc:487] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 -> Peer b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001): Couldn't send request to peer b68c081f3ce2459b8d81dc29e3d4620d. Status: Network error: Client connection negotiation failed: client connection to 127.28.186.193:45001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260806 20:14:52.817909 30229 consensus_peers.cc:487] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 -> Peer b68c081f3ce2459b8d81dc29e3d4620d (127.28.186.193:45001): Couldn't send request to peer b68c081f3ce2459b8d81dc29e3d4620d. Status: Network error: Client connection negotiation failed: client connection to 127.28.186.193:45001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:52.817917 29419 raft_consensus.cc:2258] T 78a4033e9b914dd4a00cbfeb3f6b7e66 P 6dad604353364122ae41c129a3c5cb46 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.820442 29419 tablet_replica.cc:330] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.821146 29419 raft_consensus.cc:2229] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.821638 29419 raft_consensus.cc:2258] T 3b6a183f6d714d0ba65ca608304b96d0 P 6dad604353364122ae41c129a3c5cb46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.823491 29419 tablet_replica.cc:330] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.824261 29419 raft_consensus.cc:2229] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 5 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.825167 29419 raft_consensus.cc:2258] T 89914b5b49b143a98cfe92a893ef4569 P 6dad604353364122ae41c129a3c5cb46 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.827219 29419 tablet_replica.cc:330] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.827868 29419 raft_consensus.cc:2229] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.828552 29419 raft_consensus.cc:2258] T e99ec4f4772544638e34273d315381e8 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.830706 29419 tablet_replica.cc:330] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.831368 29419 raft_consensus.cc:2229] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 3 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.832391 29419 raft_consensus.cc:2258] T 7aec719ee2834a7ba6c7d822c788ad3e P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.834705 29419 tablet_replica.cc:330] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.835341 29419 raft_consensus.cc:2229] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.835979 29419 raft_consensus.cc:2258] T 4d187995bdd1494c89d80604e6f65944 P 6dad604353364122ae41c129a3c5cb46 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.838191 29419 tablet_replica.cc:330] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.838807 29419 raft_consensus.cc:2229] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:14:52.839339 29419 raft_consensus.cc:2258] T 206cc589130d4950abb1a8cf15116498 P 6dad604353364122ae41c129a3c5cb46 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.841553 29419 tablet_replica.cc:330] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46: stopping tablet replica
I20260806 20:14:52.842123 29419 raft_consensus.cc:2229] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 4 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.843107 29419 raft_consensus.cc:2258] T 9ef940c78b6f4978a1802a7fffe371db P 6dad604353364122ae41c129a3c5cb46 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.872932 29419 tablet_server.cc:194] TabletServer@127.28.186.195:0 shutdown complete.
I20260806 20:14:52.893417 29419 master.cc:537] Master@127.28.186.254:44285 shutting down...
I20260806 20:14:52.914470 29419 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:14:52.915134 29419 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:14:52.915469 29419 tablet_replica.cc:330] T 00000000000000000000000000000000 P a432c5fe983f4d9d9723bfcbaccde844: stopping tablet replica
I20260806 20:14:52.935045 29419 master.cc:552] Master@127.28.186.254:44285 shutdown complete.
6dad604353364122ae41c129a3c5cb46 1
975e4cda6abe46838b148df041cc3b2d 4
b68c081f3ce2459b8d81dc29e3d4620d 5
6dad604353364122ae41c129a3c5cb46 3
975e4cda6abe46838b148df041cc3b2d 4
b68c081f3ce2459b8d81dc29e3d4620d 3
6dad604353364122ae41c129a3c5cb46 1
975e4cda6abe46838b148df041cc3b2d 1
b68c081f3ce2459b8d81dc29e3d4620d 8
6dad604353364122ae41c129a3c5cb46 3
975e4cda6abe46838b148df041cc3b2d 3
b68c081f3ce2459b8d81dc29e3d4620d 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29512 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260806 20:14:52.976181 29419 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.186.254:40443
I20260806 20:14:52.977305 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:14:52.983291 30579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:52.983381 30578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:52.984333 30581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:52.985388 29419 server_base.cc:1034] running on GCE node
I20260806 20:14:52.986835 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:14:52.987013 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:14:52.987149 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047292987134 us; error 0 us; skew 500 ppm
I20260806 20:14:52.987680 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:14:52.990150 29419 webserver.cc:460] Webserver started at http://127.28.186.254:40913/ using document root <none> and password file <none>
I20260806 20:14:52.990617 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:14:52.990792 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:14:52.991070 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:14:52.992276 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/master-0-root/instance:
uuid: "25d2453323cc4a96886081ec67cef6f6"
format_stamp: "Formatted at 2026-08-06 20:14:52 on dist-test-slave-49h3"
I20260806 20:14:52.996990 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 20:14:53.000348 30586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:53.001060 29419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 20:14:53.001348 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/master-0-root
uuid: "25d2453323cc4a96886081ec67cef6f6"
format_stamp: "Formatted at 2026-08-06 20:14:52 on dist-test-slave-49h3"
I20260806 20:14:53.001677 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:14:53.027714 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:14:53.028982 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:14:53.067348 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.254:40443
I20260806 20:14:53.067443 30639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.254:40443 every 8 connection(s)
I20260806 20:14:53.071231 30640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.080787 30640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Bootstrap starting.
I20260806 20:14:53.085078 30640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:53.089516 30640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: No bootstrap required, opened a new log
I20260806 20:14:53.091517 30640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d2453323cc4a96886081ec67cef6f6" member_type: VOTER }
I20260806 20:14:53.091979 30640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:53.092162 30640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d2453323cc4a96886081ec67cef6f6, State: Initialized, Role: FOLLOWER
I20260806 20:14:53.092689 30640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d2453323cc4a96886081ec67cef6f6" member_type: VOTER }
I20260806 20:14:53.093106 30640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:14:53.093298 30640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:14:53.093557 30640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:53.097384 30640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d2453323cc4a96886081ec67cef6f6" member_type: VOTER }
I20260806 20:14:53.097875 30640 leader_election.cc:302] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d2453323cc4a96886081ec67cef6f6; no voters:
I20260806 20:14:53.098948 30640 leader_election.cc:288] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:14:53.099339 30643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:53.100638 30643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [term 1 LEADER]: Becoming Leader. State: Replica: 25d2453323cc4a96886081ec67cef6f6, State: Running, Role: LEADER
I20260806 20:14:53.101298 30643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d2453323cc4a96886081ec67cef6f6" member_type: VOTER }
I20260806 20:14:53.101835 30640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:14:53.105894 30644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25d2453323cc4a96886081ec67cef6f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d2453323cc4a96886081ec67cef6f6" member_type: VOTER } }
I20260806 20:14:53.106082 30647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d2453323cc4a96886081ec67cef6f6. Latest consensus state: current_term: 1 leader_uuid: "25d2453323cc4a96886081ec67cef6f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d2453323cc4a96886081ec67cef6f6" member_type: VOTER } }
I20260806 20:14:53.106479 30644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [sys.catalog]: This master's current role is: LEADER
I20260806 20:14:53.106678 30647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6 [sys.catalog]: This master's current role is: LEADER
I20260806 20:14:53.111765 30652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:14:53.116504 30652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:14:53.117614 29419 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 20:14:53.125092 30652 catalog_manager.cc:1347] Generated new cluster ID: 2902a1a7041c40fb8f05b472d66ca016
I20260806 20:14:53.125396 30652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:14:53.160631 30652 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:14:53.162271 30652 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:14:53.178045 30652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Generated new TSK 0
I20260806 20:14:53.178705 30652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:14:53.184303 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:14:53.189936 30664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:53.191146 30665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:53.192960 29419 server_base.cc:1034] running on GCE node
W20260806 20:14:53.193385 30667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:53.194267 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:14:53.194449 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:14:53.194603 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047293194589 us; error 0 us; skew 500 ppm
I20260806 20:14:53.195111 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:14:53.197350 29419 webserver.cc:460] Webserver started at http://127.28.186.193:45009/ using document root <none> and password file <none>
I20260806 20:14:53.197857 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:14:53.198025 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:14:53.198273 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:14:53.199343 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/instance:
uuid: "42cabca48db7403b8523a3382a046043"
format_stamp: "Formatted at 2026-08-06 20:14:53 on dist-test-slave-49h3"
I20260806 20:14:53.204351 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 20:14:53.207579 30672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:53.208261 29419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 20:14:53.208524 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root
uuid: "42cabca48db7403b8523a3382a046043"
format_stamp: "Formatted at 2026-08-06 20:14:53 on dist-test-slave-49h3"
I20260806 20:14:53.208791 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:14:53.227087 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:14:53.228308 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:14:53.229806 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:14:53.232072 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:14:53.232254 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:53.232478 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:14:53.232646 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:53.273017 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.193:42937
I20260806 20:14:53.273102 30739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.193:42937 every 8 connection(s)
I20260806 20:14:53.278931 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:14:53.285653 30744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:53.286794 30745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:53.289719 29419 server_base.cc:1034] running on GCE node
W20260806 20:14:53.290566 30747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:53.291618 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:14:53.291869 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:14:53.292081 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047293292061 us; error 0 us; skew 500 ppm
I20260806 20:14:53.292624 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:14:53.293258 30740 heartbeater.cc:347] Connected to a master server at 127.28.186.254:40443
I20260806 20:14:53.293752 30740 heartbeater.cc:464] Registering TS with master...
I20260806 20:14:53.294592 30740 heartbeater.cc:511] Master 127.28.186.254:40443 requested a full tablet report, sending...
I20260806 20:14:53.295404 29419 webserver.cc:460] Webserver started at http://127.28.186.194:44007/ using document root <none> and password file <none>
I20260806 20:14:53.295912 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:14:53.296103 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:14:53.296345 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:14:53.296793 30603 ts_manager.cc:194] Registered new tserver with Master: 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:14:53.297602 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/instance:
uuid: "d4181d4d907f477f9feddc44ba3b9207"
format_stamp: "Formatted at 2026-08-06 20:14:53 on dist-test-slave-49h3"
I20260806 20:14:53.298753 30603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41550
I20260806 20:14:53.303107 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:14:53.306622 30753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:53.307327 29419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 20:14:53.307586 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root
uuid: "d4181d4d907f477f9feddc44ba3b9207"
format_stamp: "Formatted at 2026-08-06 20:14:53 on dist-test-slave-49h3"
I20260806 20:14:53.307852 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:14:53.327844 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:14:53.329052 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:14:53.330536 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:14:53.332681 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:14:53.332854 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:53.333070 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:14:53.333212 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:53.371255 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.194:42029
I20260806 20:14:53.371364 30818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.194:42029 every 8 connection(s)
I20260806 20:14:53.376206 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:14:53.382439 30822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:14:53.384524 30823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:53.386047 29419 server_base.cc:1034] running on GCE node
W20260806 20:14:53.386822 30827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:14:53.387914 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:14:53.388185 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:14:53.388365 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047293388352 us; error 0 us; skew 500 ppm
I20260806 20:14:53.388989 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:14:53.389215 30819 heartbeater.cc:347] Connected to a master server at 127.28.186.254:40443
I20260806 20:14:53.389672 30819 heartbeater.cc:464] Registering TS with master...
I20260806 20:14:53.390612 30819 heartbeater.cc:511] Master 127.28.186.254:40443 requested a full tablet report, sending...
I20260806 20:14:53.391992 29419 webserver.cc:460] Webserver started at http://127.28.186.195:40681/ using document root <none> and password file <none>
I20260806 20:14:53.392550 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:14:53.392726 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:14:53.392795 30603 ts_manager.cc:194] Registered new tserver with Master: d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:53.392983 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:14:53.394268 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/instance:
uuid: "58d59beb597f4ed99d84d102ddb41ff5"
format_stamp: "Formatted at 2026-08-06 20:14:53 on dist-test-slave-49h3"
I20260806 20:14:53.394418 30603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41560
I20260806 20:14:53.399026 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 20:14:53.402436 30831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:53.403239 29419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 20:14:53.403512 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root
uuid: "58d59beb597f4ed99d84d102ddb41ff5"
format_stamp: "Formatted at 2026-08-06 20:14:53 on dist-test-slave-49h3"
I20260806 20:14:53.403791 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:14:53.428011 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:14:53.429241 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:14:53.430712 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:14:53.432950 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:14:53.433153 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:14:53.433370 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:14:53.433531 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 20:14:53.473124 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.195:41735
I20260806 20:14:53.473316 30897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.195:41735 every 8 connection(s)
I20260806 20:14:53.488054 30898 heartbeater.cc:347] Connected to a master server at 127.28.186.254:40443
I20260806 20:14:53.488444 30898 heartbeater.cc:464] Registering TS with master...
I20260806 20:14:53.489123 30898 heartbeater.cc:511] Master 127.28.186.254:40443 requested a full tablet report, sending...
I20260806 20:14:53.491160 30603 ts_manager.cc:194] Registered new tserver with Master: 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:53.491840 29419 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015762567s
I20260806 20:14:53.492726 30603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41564
I20260806 20:14:53.494045 29419 test_util.cc:268] Using random seed: -1971645687
I20260806 20:14:53.519411 30603 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41566:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:14:53.525764 30603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:14:53.725590 30702 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba15abedb154a46b9f13784776fbeae (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 20:14:53.726940 30702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba15abedb154a46b9f13784776fbeae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.728456 30693 tablet_service.cc:1467] Processing CreateTablet for tablet 47c09420eb1e4f3692c165f76cd1a7d4 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 20:14:53.729590 30860 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba15abedb154a46b9f13784776fbeae (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 20:14:53.730943 30860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba15abedb154a46b9f13784776fbeae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.730602 30699 tablet_service.cc:1467] Processing CreateTablet for tablet e1485b55849242feae0e8b2b3b28307f (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 20:14:53.731446 30856 tablet_service.cc:1467] Processing CreateTablet for tablet 28d6e84eb2874b2b9dca3387b3d98eff (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 20:14:53.731947 30699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1485b55849242feae0e8b2b3b28307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.733203 30855 tablet_service.cc:1467] Processing CreateTablet for tablet dc53de31500c4908b8ec8ee3a0f7498c (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 20:14:53.734550 30855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc53de31500c4908b8ec8ee3a0f7498c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.734586 30859 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb9334179bc40b1b3868fc4773af362 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 20:14:53.735904 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb9334179bc40b1b3868fc4773af362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.735793 30695 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8d5f5ccc46476aac7a641abae6bdfa (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 20:14:53.737063 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8d5f5ccc46476aac7a641abae6bdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.737383 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba15abedb154a46b9f13784776fbeae (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 20:14:53.738651 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba15abedb154a46b9f13784776fbeae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.738559 30696 tablet_service.cc:1467] Processing CreateTablet for tablet dc53de31500c4908b8ec8ee3a0f7498c (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 20:14:53.739410 30857 tablet_service.cc:1467] Processing CreateTablet for tablet f92e9dc9db864f0b8e238a39be31b12e (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 20:14:53.739830 30696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc53de31500c4908b8ec8ee3a0f7498c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.740679 30857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92e9dc9db864f0b8e238a39be31b12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.728058 30694 tablet_service.cc:1467] Processing CreateTablet for tablet eca12ca38e014311bbb9373e14cc903b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 20:14:53.742622 30700 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb9334179bc40b1b3868fc4773af362 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 20:14:53.743806 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca12ca38e014311bbb9373e14cc903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.744436 30697 tablet_service.cc:1467] Processing CreateTablet for tablet 28d6e84eb2874b2b9dca3387b3d98eff (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 20:14:53.744629 30850 tablet_service.cc:1467] Processing CreateTablet for tablet 26a3d1fc21e040c09103cfcf9e539659 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 20:14:53.742421 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 47c09420eb1e4f3692c165f76cd1a7d4 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 20:14:53.745961 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a3d1fc21e040c09103cfcf9e539659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.746690 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c09420eb1e4f3692c165f76cd1a7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.746876 30692 tablet_service.cc:1467] Processing CreateTablet for tablet 26a3d1fc21e040c09103cfcf9e539659 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 20:14:53.747193 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb9334179bc40b1b3868fc4773af362 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 20:14:53.748404 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb9334179bc40b1b3868fc4773af362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.749182 30853 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8d5f5ccc46476aac7a641abae6bdfa (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 20:14:53.750492 30853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8d5f5ccc46476aac7a641abae6bdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.750397 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d6e84eb2874b2b9dca3387b3d98eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.742430 30858 tablet_service.cc:1467] Processing CreateTablet for tablet e1485b55849242feae0e8b2b3b28307f (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 20:14:53.748358 30852 tablet_service.cc:1467] Processing CreateTablet for tablet eca12ca38e014311bbb9373e14cc903b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 20:14:53.753428 30858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1485b55849242feae0e8b2b3b28307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.756665 30780 tablet_service.cc:1467] Processing CreateTablet for tablet e1485b55849242feae0e8b2b3b28307f (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 20:14:53.757850 30856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d6e84eb2874b2b9dca3387b3d98eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.758014 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1485b55849242feae0e8b2b3b28307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.734634 30698 tablet_service.cc:1467] Processing CreateTablet for tablet f92e9dc9db864f0b8e238a39be31b12e (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 20:14:53.761698 30779 tablet_service.cc:1467] Processing CreateTablet for tablet f92e9dc9db864f0b8e238a39be31b12e (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 20:14:53.763088 30779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92e9dc9db864f0b8e238a39be31b12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.765836 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca12ca38e014311bbb9373e14cc903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.765647 30778 tablet_service.cc:1467] Processing CreateTablet for tablet 28d6e84eb2874b2b9dca3387b3d98eff (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 20:14:53.767156 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d6e84eb2874b2b9dca3387b3d98eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.768848 30773 tablet_service.cc:1467] Processing CreateTablet for tablet 26a3d1fc21e040c09103cfcf9e539659 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 20:14:53.770255 30773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a3d1fc21e040c09103cfcf9e539659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.773567 30777 tablet_service.cc:1467] Processing CreateTablet for tablet dc53de31500c4908b8ec8ee3a0f7498c (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 20:14:53.758438 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb9334179bc40b1b3868fc4773af362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.775671 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc53de31500c4908b8ec8ee3a0f7498c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.776121 30693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c09420eb1e4f3692c165f76cd1a7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.777314 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92e9dc9db864f0b8e238a39be31b12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.778581 30774 tablet_service.cc:1467] Processing CreateTablet for tablet 47c09420eb1e4f3692c165f76cd1a7d4 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 20:14:53.779883 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c09420eb1e4f3692c165f76cd1a7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.782563 30775 tablet_service.cc:1467] Processing CreateTablet for tablet eca12ca38e014311bbb9373e14cc903b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 20:14:53.783906 30775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca12ca38e014311bbb9373e14cc903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.786643 30776 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8d5f5ccc46476aac7a641abae6bdfa (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 20:14:53.788905 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8d5f5ccc46476aac7a641abae6bdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.819020 30698 tablet_service.cc:1467] Processing CreateTablet for tablet e39c6ca59dfb42bda66c72b0bcf52faa (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 20:14:53.871279 30781 tablet_service.cc:1467] Processing CreateTablet for tablet e39c6ca59dfb42bda66c72b0bcf52faa (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 20:14:53.872671 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39c6ca59dfb42bda66c72b0bcf52faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.877581 30692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26a3d1fc21e040c09103cfcf9e539659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.916756 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 6868aeec902a4311a782f4ef5ab064f8 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 20:14:53.947739 30914 tablet_bootstrap.cc:492] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:53.954871 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39c6ca59dfb42bda66c72b0bcf52faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.959995 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 6868aeec902a4311a782f4ef5ab064f8 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 20:14:53.961169 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6868aeec902a4311a782f4ef5ab064f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.963011 30779 tablet_service.cc:1467] Processing CreateTablet for tablet a439c5debb41440c913a50da91e6467f (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 20:14:53.964843 30913 tablet_bootstrap.cc:492] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:53.966123 30860 tablet_service.cc:1467] Processing CreateTablet for tablet 672b273ef36e4c6eb6873cc61471cb53 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 20:14:53.967418 30860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b273ef36e4c6eb6873cc61471cb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.968895 30855 tablet_service.cc:1467] Processing CreateTablet for tablet 6868aeec902a4311a782f4ef5ab064f8 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 20:14:53.969894 30915 tablet_bootstrap.cc:492] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:53.970242 30855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6868aeec902a4311a782f4ef5ab064f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.978153 30698 tablet_service.cc:1467] Processing CreateTablet for tablet 67c52f63d3e8412bbd0e51fe5b81568b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 20:14:53.979451 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c52f63d3e8412bbd0e51fe5b81568b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.979238 30858 tablet_service.cc:1467] Processing CreateTablet for tablet a439c5debb41440c913a50da91e6467f (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 20:14:53.982049 30858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a439c5debb41440c913a50da91e6467f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.981813 30699 tablet_service.cc:1467] Processing CreateTablet for tablet a439c5debb41440c913a50da91e6467f (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 20:14:53.993993 30777 tablet_service.cc:1467] Processing CreateTablet for tablet 67c52f63d3e8412bbd0e51fe5b81568b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 20:14:53.994802 30780 tablet_service.cc:1467] Processing CreateTablet for tablet 672b273ef36e4c6eb6873cc61471cb53 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 20:14:53.985250 30695 tablet_service.cc:1467] Processing CreateTablet for tablet 672b273ef36e4c6eb6873cc61471cb53 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 20:14:53.994871 30851 tablet_service.cc:1467] Processing CreateTablet for tablet e39c6ca59dfb42bda66c72b0bcf52faa (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 20:14:53.995594 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c52f63d3e8412bbd0e51fe5b81568b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.996461 30773 tablet_service.cc:1467] Processing CreateTablet for tablet e0607668d3924d5cba6acd89b2cdc25f (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 20:14:53.997671 30773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0607668d3924d5cba6acd89b2cdc25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.986858 30856 tablet_service.cc:1467] Processing CreateTablet for tablet 67c52f63d3e8412bbd0e51fe5b81568b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 20:14:54.000211 30779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a439c5debb41440c913a50da91e6467f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.994500 30699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a439c5debb41440c913a50da91e6467f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.001772 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b273ef36e4c6eb6873cc61471cb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:53.984468 30914 tablet_bootstrap.cc:654] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.003604 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b273ef36e4c6eb6873cc61471cb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.009721 30852 tablet_service.cc:1467] Processing CreateTablet for tablet e0607668d3924d5cba6acd89b2cdc25f (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 20:14:53.998492 30697 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3393f94b2649378cb19b57fecfb21e (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 20:14:54.017192 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3393f94b2649378cb19b57fecfb21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.018028 30696 tablet_service.cc:1467] Processing CreateTablet for tablet f0cdb10a40ac4d0bbcfc4ed368b6dc43 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 20:14:54.019310 30696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0cdb10a40ac4d0bbcfc4ed368b6dc43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.021607 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6868aeec902a4311a782f4ef5ab064f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.025053 30776 tablet_service.cc:1467] Processing CreateTablet for tablet 7e6251f4f1aa48ca869b81256b9fd89a (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 20:14:54.025830 30850 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3393f94b2649378cb19b57fecfb21e (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 20:14:54.026561 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6251f4f1aa48ca869b81256b9fd89a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.027200 30857 tablet_service.cc:1467] Processing CreateTablet for tablet f0cdb10a40ac4d0bbcfc4ed368b6dc43 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 20:14:54.029289 30853 tablet_service.cc:1467] Processing CreateTablet for tablet 7e6251f4f1aa48ca869b81256b9fd89a (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 20:14:54.030861 30913 tablet_bootstrap.cc:654] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.037603 30702 tablet_service.cc:1467] Processing CreateTablet for tablet 7e6251f4f1aa48ca869b81256b9fd89a (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 20:14:54.038944 30702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6251f4f1aa48ca869b81256b9fd89a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.040654 30915 tablet_bootstrap.cc:654] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:53.976832 30700 tablet_service.cc:1467] Processing CreateTablet for tablet e0607668d3924d5cba6acd89b2cdc25f (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 20:14:54.042447 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0607668d3924d5cba6acd89b2cdc25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.048820 30859 tablet_service.cc:1467] Processing CreateTablet for tablet b7665c0d8832432f8948e82d823498fd (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 20:14:54.049685 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39c6ca59dfb42bda66c72b0bcf52faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.053988 30692 tablet_service.cc:1467] Processing CreateTablet for tablet 88608755c61d42ec8ccc0c3c8e536bd6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 20:14:54.065371 30856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c52f63d3e8412bbd0e51fe5b81568b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.065785 30692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88608755c61d42ec8ccc0c3c8e536bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.067669 30778 tablet_service.cc:1467] Processing CreateTablet for tablet b7665c0d8832432f8948e82d823498fd (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 20:14:54.055541 30698 tablet_service.cc:1467] Processing CreateTablet for tablet 33e2a287c4de469eb5365fa7cdc6b409 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 20:14:54.068878 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7665c0d8832432f8948e82d823498fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.069175 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e2a287c4de469eb5365fa7cdc6b409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.069738 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7665c0d8832432f8948e82d823498fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.070336 30857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0cdb10a40ac4d0bbcfc4ed368b6dc43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.045053 30774 tablet_service.cc:1467] Processing CreateTablet for tablet 0f3393f94b2649378cb19b57fecfb21e (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 20:14:54.074944 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3393f94b2649378cb19b57fecfb21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.077749 30853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6251f4f1aa48ca869b81256b9fd89a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.086733 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3393f94b2649378cb19b57fecfb21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.058058 30694 tablet_service.cc:1467] Processing CreateTablet for tablet b7665c0d8832432f8948e82d823498fd (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 20:14:54.060730 30775 tablet_service.cc:1467] Processing CreateTablet for tablet f0cdb10a40ac4d0bbcfc4ed368b6dc43 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 20:14:54.061369 30697 tablet_service.cc:1467] Processing CreateTablet for tablet c0005c7c8586414a81bd94a504a598c9 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 20:14:54.100797 30775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0cdb10a40ac4d0bbcfc4ed368b6dc43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.108554 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7665c0d8832432f8948e82d823498fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.109421 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0005c7c8586414a81bd94a504a598c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:14:54.113266 30655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:14:54.126827 30777 tablet_service.cc:1467] Processing CreateTablet for tablet 88608755c61d42ec8ccc0c3c8e536bd6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 20:14:54.128227 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88608755c61d42ec8ccc0c3c8e536bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.131779 30692 tablet_service.cc:1467] Processing CreateTablet for tablet a5e69bf76cc845c8822b4cfab14eac47 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 20:14:54.134958 30857 tablet_service.cc:1467] Processing CreateTablet for tablet 33e2a287c4de469eb5365fa7cdc6b409 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 20:14:54.137482 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0607668d3924d5cba6acd89b2cdc25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.138224 30855 tablet_service.cc:1467] Processing CreateTablet for tablet 88608755c61d42ec8ccc0c3c8e536bd6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 20:14:54.138258 30773 tablet_service.cc:1467] Processing CreateTablet for tablet c0005c7c8586414a81bd94a504a598c9 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 20:14:54.145556 30913 tablet_bootstrap.cc:492] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:54.146162 30913 ts_tablet_manager.cc:1397] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.182s user 0.011s sys 0.004s
I20260806 20:14:54.146364 30773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0005c7c8586414a81bd94a504a598c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.150198 30692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e69bf76cc845c8822b4cfab14eac47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.151571 30914 tablet_bootstrap.cc:492] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:54.152067 30914 ts_tablet_manager.cc:1397] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.205s user 0.009s sys 0.004s
I20260806 20:14:54.154919 30914 raft_consensus.cc:348] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.155658 30914 raft_consensus.cc:374] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:54.155943 30914 raft_consensus.cc:729] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:54.156381 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 3070a7930e6d47cd8776055aab6539a3 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 20:14:54.156773 30914 consensus_queue.cc:260] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.172262 30699 tablet_service.cc:1467] Processing CreateTablet for tablet f53758bb873349479f86e0abf34c738b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 20:14:54.163254 30780 tablet_service.cc:1467] Processing CreateTablet for tablet 0911118ae8e247aca188955eb02c8728 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 20:14:54.164989 30695 tablet_service.cc:1467] Processing CreateTablet for tablet 0911118ae8e247aca188955eb02c8728 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 20:14:54.165987 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 33e2a287c4de469eb5365fa7cdc6b409 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 20:14:54.149101 30913 raft_consensus.cc:348] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.138387 30693 tablet_service.cc:1467] Processing CreateTablet for tablet 3070a7930e6d47cd8776055aab6539a3 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 20:14:54.175024 30699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53758bb873349479f86e0abf34c738b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.180626 30913 raft_consensus.cc:374] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:54.182292 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3070a7930e6d47cd8776055aab6539a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.189438 30898 heartbeater.cc:503] Master 127.28.186.254:40443 was elected leader, sending a full tablet report...
I20260806 20:14:54.191159 30779 tablet_service.cc:1467] Processing CreateTablet for tablet a5e69bf76cc845c8822b4cfab14eac47 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 20:14:54.196599 30914 ts_tablet_manager.cc:1428] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.044s user 0.009s sys 0.000s
I20260806 20:14:54.196965 30855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88608755c61d42ec8ccc0c3c8e536bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.197654 30914 tablet_bootstrap.cc:492] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:54.200493 30857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e2a287c4de469eb5365fa7cdc6b409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.206336 30860 tablet_service.cc:1467] Processing CreateTablet for tablet c0005c7c8586414a81bd94a504a598c9 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 20:14:54.207690 30860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0005c7c8586414a81bd94a504a598c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.208388 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e2a287c4de469eb5365fa7cdc6b409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.185282 30913 raft_consensus.cc:729] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:54.211841 30779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e69bf76cc845c8822b4cfab14eac47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.213250 30915 tablet_bootstrap.cc:492] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:54.213757 30915 ts_tablet_manager.cc:1397] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.244s user 0.007s sys 0.011s
I20260806 20:14:54.214469 30913 consensus_queue.cc:260] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.216337 30915 raft_consensus.cc:348] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.217108 30915 raft_consensus.cc:374] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:54.217531 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0911118ae8e247aca188955eb02c8728. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.204896 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 3070a7930e6d47cd8776055aab6539a3 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 20:14:54.216988 30859 tablet_service.cc:1467] Processing CreateTablet for tablet a5e69bf76cc845c8822b4cfab14eac47 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 20:14:54.222564 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 47b5901783b8423c9e494c131b0a64e9 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
W20260806 20:14:54.228494 30899 tablet.cc:2369] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:14:54.217409 30915 raft_consensus.cc:729] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:54.232256 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0911118ae8e247aca188955eb02c8728. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.230415 30693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3070a7930e6d47cd8776055aab6539a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.238652 30698 tablet_service.cc:1467] Processing CreateTablet for tablet bf7304ca0290443b884297d0e8744607 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 20:14:54.239496 30702 tablet_service.cc:1467] Processing CreateTablet for tablet 3e24eef6446040b8abbe977d4dd5a3ca (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 20:14:54.240393 30915 consensus_queue.cc:260] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.240897 30775 tablet_service.cc:1467] Processing CreateTablet for tablet 47b5901783b8423c9e494c131b0a64e9 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 20:14:54.242220 30775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b5901783b8423c9e494c131b0a64e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.247613 30778 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec60d6f5b6c486b80781ad3cb85a06b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 20:14:54.240398 30776 tablet_service.cc:1467] Processing CreateTablet for tablet f53758bb873349479f86e0abf34c738b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 20:14:54.248860 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec60d6f5b6c486b80781ad3cb85a06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.241003 30696 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec60d6f5b6c486b80781ad3cb85a06b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 20:14:54.249791 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53758bb873349479f86e0abf34c738b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.250485 30774 tablet_service.cc:1467] Processing CreateTablet for tablet 95c5938e508b4075b878ae1f5ea726d6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 20:14:54.268795 30819 heartbeater.cc:503] Master 127.28.186.254:40443 was elected leader, sending a full tablet report...
I20260806 20:14:54.269495 30920 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:54.269956 30920 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.270767 30858 tablet_service.cc:1467] Processing CreateTablet for tablet 0911118ae8e247aca188955eb02c8728 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 20:14:54.272449 30920 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:54.264832 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3070a7930e6d47cd8776055aab6539a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.285081 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b5901783b8423c9e494c131b0a64e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.287451 30777 tablet_service.cc:1467] Processing CreateTablet for tablet bf7304ca0290443b884297d0e8744607 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 20:14:54.290160 30853 tablet_service.cc:1467] Processing CreateTablet for tablet f53758bb873349479f86e0abf34c738b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 20:14:54.307153 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7304ca0290443b884297d0e8744607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.307950 30740 heartbeater.cc:503] Master 127.28.186.254:40443 was elected leader, sending a full tablet report...
I20260806 20:14:54.309701 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c5938e508b4075b878ae1f5ea726d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.322531 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e69bf76cc845c8822b4cfab14eac47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.335119 30773 tablet_service.cc:1467] Processing CreateTablet for tablet 3e24eef6446040b8abbe977d4dd5a3ca (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 20:14:54.335484 30779 tablet_service.cc:1467] Processing CreateTablet for tablet 24eb40e43f13412e8f080b6b6ca292de (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 20:14:54.336517 30773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e24eef6446040b8abbe977d4dd5a3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.337137 30913 ts_tablet_manager.cc:1428] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.191s user 0.010s sys 0.011s
I20260806 20:14:54.338179 30913 tablet_bootstrap.cc:492] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:54.339190 30775 tablet_service.cc:1467] Processing CreateTablet for tablet 01fd0995e74a449090ea2b00062f625e (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 20:14:54.340430 30775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fd0995e74a449090ea2b00062f625e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.344740 30914 tablet_bootstrap.cc:654] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.351758 30856 tablet_service.cc:1467] Processing CreateTablet for tablet 47b5901783b8423c9e494c131b0a64e9 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 20:14:54.354463 30855 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec60d6f5b6c486b80781ad3cb85a06b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 20:14:54.354956 30857 tablet_service.cc:1467] Processing CreateTablet for tablet 95c5938e508b4075b878ae1f5ea726d6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 20:14:54.355525 30853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53758bb873349479f86e0abf34c738b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.362488 30857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c5938e508b4075b878ae1f5ea726d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.364101 30697 tablet_service.cc:1467] Processing CreateTablet for tablet 24eb40e43f13412e8f080b6b6ca292de (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 20:14:54.365341 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eb40e43f13412e8f080b6b6ca292de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.366918 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 48c353d311a64e628bfc1af985e1d8ba (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 20:14:54.368948 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7304ca0290443b884297d0e8744607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.373785 30856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b5901783b8423c9e494c131b0a64e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.377717 30858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0911118ae8e247aca188955eb02c8728. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.378067 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 01fd0995e74a449090ea2b00062f625e (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 20:14:54.379274 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fd0995e74a449090ea2b00062f625e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.381989 30702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e24eef6446040b8abbe977d4dd5a3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.356382 30700 tablet_service.cc:1467] Processing CreateTablet for tablet 95c5938e508b4075b878ae1f5ea726d6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 20:14:54.384608 30782 tablet_service.cc:1467] Processing CreateTablet for tablet aa7cfe759ca64ca89a0529c2d3da5bf7 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 20:14:54.385758 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c5938e508b4075b878ae1f5ea726d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.386767 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:14:54.388427 30756 leader_election.cc:341] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.389791 30696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec60d6f5b6c486b80781ad3cb85a06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.390352 30855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec60d6f5b6c486b80781ad3cb85a06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.403690 30913 tablet_bootstrap.cc:654] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.404793 30779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eb40e43f13412e8f080b6b6ca292de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.407656 30857 tablet_service.cc:1467] Processing CreateTablet for tablet bf7304ca0290443b884297d0e8744607 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 20:14:54.408494 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c353d311a64e628bfc1af985e1d8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.411739 30857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf7304ca0290443b884297d0e8744607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.412194 30915 ts_tablet_manager.cc:1428] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.198s user 0.020s sys 0.002s
I20260806 20:14:54.413260 30915 tablet_bootstrap.cc:492] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:54.417378 30699 tablet_service.cc:1467] Processing CreateTablet for tablet ec226f72b38a49529b88339beb36af46 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 20:14:54.421662 30699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec226f72b38a49529b88339beb36af46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.423043 30850 tablet_service.cc:1467] Processing CreateTablet for tablet 24eb40e43f13412e8f080b6b6ca292de (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 20:14:54.424314 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eb40e43f13412e8f080b6b6ca292de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.419883 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7cfe759ca64ca89a0529c2d3da5bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.427436 30773 tablet_service.cc:1467] Processing CreateTablet for tablet cd962fc44d164b8b80175c097e321044 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 20:14:54.428836 30773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd962fc44d164b8b80175c097e321044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.433384 30695 tablet_service.cc:1467] Processing CreateTablet for tablet aa7cfe759ca64ca89a0529c2d3da5bf7 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 20:14:54.434772 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7cfe759ca64ca89a0529c2d3da5bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.389353 30692 tablet_service.cc:1467] Processing CreateTablet for tablet 48c353d311a64e628bfc1af985e1d8ba (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 20:14:54.438498 30692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c353d311a64e628bfc1af985e1d8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.443567 30776 tablet_service.cc:1467] Processing CreateTablet for tablet ec226f72b38a49529b88339beb36af46 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 20:14:54.445042 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec226f72b38a49529b88339beb36af46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.437582 30914 tablet_bootstrap.cc:492] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:54.446817 30914 ts_tablet_manager.cc:1397] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.249s user 0.015s sys 0.017s
I20260806 20:14:54.450969 30852 tablet_service.cc:1467] Processing CreateTablet for tablet 3e24eef6446040b8abbe977d4dd5a3ca (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 20:14:54.452394 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e24eef6446040b8abbe977d4dd5a3ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:14:54.477756 30590 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 51643.4us
I20260806 20:14:54.486104 30914 raft_consensus.cc:348] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.486888 30914 raft_consensus.cc:374] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:54.487185 30914 raft_consensus.cc:729] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:54.489140 30915 tablet_bootstrap.cc:654] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.526357 30698 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0db3302c934290a93017bbfb29db44 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 20:14:54.527717 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0db3302c934290a93017bbfb29db44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.496567 30860 tablet_service.cc:1467] Processing CreateTablet for tablet 01fd0995e74a449090ea2b00062f625e (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 20:14:54.528740 30860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fd0995e74a449090ea2b00062f625e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.493737 30914 consensus_queue.cc:260] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.538060 30697 tablet_service.cc:1467] Processing CreateTablet for tablet 616aae135c70498da00a9e8a1a61b1cd (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 20:14:54.539323 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616aae135c70498da00a9e8a1a61b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.501363 30702 tablet_service.cc:1467] Processing CreateTablet for tablet a1066045a99d4a1da0fff49e2b8bbe30 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 20:14:54.542685 30702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1066045a99d4a1da0fff49e2b8bbe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.542845 30914 ts_tablet_manager.cc:1428] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.096s user 0.006s sys 0.001s
I20260806 20:14:54.543761 30914 tablet_bootstrap.cc:492] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:54.545020 30919 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:54.545490 30919 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.547461 30913 tablet_bootstrap.cc:492] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:54.547943 30919 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:54.549569 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 942abb68076e47a0a24672bb92649892 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 20:14:54.551476 30699 tablet_service.cc:1467] Processing CreateTablet for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 20:14:54.557395 30850 tablet_service.cc:1467] Processing CreateTablet for tablet ec226f72b38a49529b88339beb36af46 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 20:14:54.558969 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec226f72b38a49529b88339beb36af46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.498034 30693 tablet_service.cc:1467] Processing CreateTablet for tablet cd962fc44d164b8b80175c097e321044 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 20:14:54.503633 30780 tablet_service.cc:1467] Processing CreateTablet for tablet a1066045a99d4a1da0fff49e2b8bbe30 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 20:14:54.510087 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:54.516417 30853 tablet_service.cc:1467] Processing CreateTablet for tablet 48c353d311a64e628bfc1af985e1d8ba (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 20:14:54.560088 30913 ts_tablet_manager.cc:1397] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.210s user 0.011s sys 0.000s
I20260806 20:14:54.560968 30853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c353d311a64e628bfc1af985e1d8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.562865 30913 raft_consensus.cc:348] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.563603 30913 raft_consensus.cc:374] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:54.563885 30913 raft_consensus.cc:729] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:54.564297 30857 tablet_service.cc:1467] Processing CreateTablet for tablet aa7cfe759ca64ca89a0529c2d3da5bf7 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 20:14:54.564579 30913 consensus_queue.cc:260] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.565583 30857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7cfe759ca64ca89a0529c2d3da5bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.570156 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1066045a99d4a1da0fff49e2b8bbe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.574333 30774 tablet_service.cc:1467] Processing CreateTablet for tablet 616aae135c70498da00a9e8a1a61b1cd (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 20:14:54.575716 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616aae135c70498da00a9e8a1a61b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.575791 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 942abb68076e47a0a24672bb92649892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.562886 30700 tablet_service.cc:1467] Processing CreateTablet for tablet 761e8328b9d247fcb1d00f0aeff373f0 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 20:14:54.516820 30777 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0db3302c934290a93017bbfb29db44 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 20:14:54.581382 30858 tablet_service.cc:1467] Processing CreateTablet for tablet cd962fc44d164b8b80175c097e321044 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 20:14:54.582708 30858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd962fc44d164b8b80175c097e321044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.584643 30698 tablet_service.cc:1467] Processing CreateTablet for tablet cc6e5d61a5694983a73c242e91257d86 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 20:14:54.585937 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6e5d61a5694983a73c242e91257d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.587765 30921 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:54.588219 30921 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.590723 30921 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:54.608598 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761e8328b9d247fcb1d00f0aeff373f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.609278 30697 tablet_service.cc:1467] Processing CreateTablet for tablet b32e68e22f574e679ad1e327caa342f6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 20:14:54.622630 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 78bef7be8f95499c84c37311213c95b6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 20:14:54.610740 30699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.627750 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32e68e22f574e679ad1e327caa342f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.628336 30859 tablet_service.cc:1467] Processing CreateTablet for tablet 942abb68076e47a0a24672bb92649892 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 20:14:54.629617 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 942abb68076e47a0a24672bb92649892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.631094 30913 ts_tablet_manager.cc:1428] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.071s user 0.000s sys 0.006s
I20260806 20:14:54.632004 30913 tablet_bootstrap.cc:492] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:54.633074 30851 tablet_service.cc:1467] Processing CreateTablet for tablet a1066045a99d4a1da0fff49e2b8bbe30 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 20:14:54.624253 30773 tablet_service.cc:1467] Processing CreateTablet for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 20:14:54.634315 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1066045a99d4a1da0fff49e2b8bbe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.638599 30693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd962fc44d164b8b80175c097e321044. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.641680 30914 tablet_bootstrap.cc:654] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.645352 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0db3302c934290a93017bbfb29db44. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:14:54.665899 30757 leader_election.cc:341] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.666435 30757 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:54.667368 30922 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:54.670382 30855 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0db3302c934290a93017bbfb29db44 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 20:14:54.673413 30855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0db3302c934290a93017bbfb29db44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.683445 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bef7be8f95499c84c37311213c95b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.698498 30922 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:54.699025 30922 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.702447 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:54.701277 30922 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:54.704118 30919 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:54.704761 30919 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.705642 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:54.706821 30919 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:14:54.704169 30757 leader_election.cc:341] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:14:54.707716 30756 leader_election.cc:341] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.708318 30756 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:54.709359 30922 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:54.712826 30914 tablet_bootstrap.cc:492] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:54.714747 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 78bef7be8f95499c84c37311213c95b6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 20:14:54.716388 30853 tablet_service.cc:1467] Processing CreateTablet for tablet 616aae135c70498da00a9e8a1a61b1cd (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 20:14:54.710229 30778 tablet_service.cc:1467] Processing CreateTablet for tablet 942abb68076e47a0a24672bb92649892 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 20:14:54.726444 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:54.736075 30773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:14:54.736436 30835 leader_election.cc:341] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.738313 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:54.739702 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 942abb68076e47a0a24672bb92649892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.739332 30781 tablet_service.cc:1467] Processing CreateTablet for tablet cc6e5d61a5694983a73c242e91257d86 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
W20260806 20:14:54.745394 30835 leader_election.cc:341] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.746470 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6e5d61a5694983a73c242e91257d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.746974 30913 tablet_bootstrap.cc:654] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.732478 30914 ts_tablet_manager.cc:1397] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.189s user 0.015s sys 0.002s
I20260806 20:14:54.752381 30775 tablet_service.cc:1467] Processing CreateTablet for tablet 761e8328b9d247fcb1d00f0aeff373f0 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 20:14:54.753808 30775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761e8328b9d247fcb1d00f0aeff373f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.756892 30852 tablet_service.cc:1467] Processing CreateTablet for tablet 78bef7be8f95499c84c37311213c95b6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 20:14:54.758769 30692 tablet_service.cc:1467] Processing CreateTablet for tablet 7c541d91c0c4487c9a76c6fc6f81a95a (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 20:14:54.760059 30692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c541d91c0c4487c9a76c6fc6f81a95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.762476 30856 tablet_service.cc:1467] Processing CreateTablet for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 20:14:54.763527 30853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616aae135c70498da00a9e8a1a61b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.764685 30914 raft_consensus.cc:348] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.767155 30695 tablet_service.cc:1467] Processing CreateTablet for tablet b060012888ff4d76b254b43d1f1959bb (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 20:14:54.767493 30702 tablet_service.cc:1467] Processing CreateTablet for tablet 9f32336173df4611829be37d396e86ba (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 20:14:54.768082 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bef7be8f95499c84c37311213c95b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.792393 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:54.771358 30860 tablet_service.cc:1467] Processing CreateTablet for tablet 761e8328b9d247fcb1d00f0aeff373f0 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 20:14:54.771893 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bef7be8f95499c84c37311213c95b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.793615 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:14:54.795317 30834 leader_election.cc:341] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.795753 30834 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:54.796988 30860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761e8328b9d247fcb1d00f0aeff373f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:14:54.796950 30834 leader_election.cc:341] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.797608 30919 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:54.798300 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:54.799211 30919 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:54.771692 30857 tablet_service.cc:1467] Processing CreateTablet for tablet cc6e5d61a5694983a73c242e91257d86 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 20:14:54.802062 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:14:54.803508 30676 leader_election.cc:341] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.805492 30850 tablet_service.cc:1467] Processing CreateTablet for tablet b32e68e22f574e679ad1e327caa342f6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 20:14:54.779227 30914 raft_consensus.cc:374] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:54.806913 30914 raft_consensus.cc:729] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:54.812196 30700 tablet_service.cc:1467] Processing CreateTablet for tablet e0c1793b412c4608b1e004525a6dd887 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 20:14:54.813717 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:54.788705 30774 tablet_service.cc:1467] Processing CreateTablet for tablet 7c541d91c0c4487c9a76c6fc6f81a95a (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 20:14:54.814761 30780 tablet_service.cc:1467] Processing CreateTablet for tablet b32e68e22f574e679ad1e327caa342f6 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
W20260806 20:14:54.815227 30676 leader_election.cc:341] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.815733 30676 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:54.785008 30696 tablet_service.cc:1467] Processing CreateTablet for tablet 412dab7986704f48be75b1fdbdccc9fa (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 20:14:54.816569 30921 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:54.816008 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c541d91c0c4487c9a76c6fc6f81a95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.802371 30857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6e5d61a5694983a73c242e91257d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.822603 30915 tablet_bootstrap.cc:492] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:54.823099 30915 ts_tablet_manager.cc:1397] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.410s user 0.016s sys 0.037s
I20260806 20:14:54.821127 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 7c541d91c0c4487c9a76c6fc6f81a95a (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 20:14:54.825623 30914 consensus_queue.cc:260] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.827329 30852 tablet_service.cc:1467] Processing CreateTablet for tablet b060012888ff4d76b254b43d1f1959bb (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 20:14:54.828846 30914 ts_tablet_manager.cc:1428] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.079s user 0.003s sys 0.003s
I20260806 20:14:54.829651 30915 raft_consensus.cc:348] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.830385 30915 raft_consensus.cc:374] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:54.830662 30915 raft_consensus.cc:729] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:54.831310 30915 consensus_queue.cc:260] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.831900 30693 tablet_service.cc:1467] Processing CreateTablet for tablet 15c9bdefd8144ce787ecb60a54ea1450 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 20:14:54.833561 30915 ts_tablet_manager.cc:1428] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260806 20:14:54.835294 30915 tablet_bootstrap.cc:492] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:54.838017 30856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.839663 30913 tablet_bootstrap.cc:492] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:54.840134 30913 ts_tablet_manager.cc:1397] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.208s user 0.015s sys 0.013s
I20260806 20:14:54.845836 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32e68e22f574e679ad1e327caa342f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.847003 30913 raft_consensus.cc:348] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.847797 30913 raft_consensus.cc:374] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:54.848095 30913 raft_consensus.cc:729] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:54.848908 30913 consensus_queue.cc:260] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.851234 30780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32e68e22f574e679ad1e327caa342f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.860323 30776 tablet_service.cc:1467] Processing CreateTablet for tablet 9f32336173df4611829be37d396e86ba (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 20:14:54.861027 30777 tablet_service.cc:1467] Processing CreateTablet for tablet 412dab7986704f48be75b1fdbdccc9fa (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 20:14:54.861665 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f32336173df4611829be37d396e86ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.862417 30782 tablet_service.cc:1467] Processing CreateTablet for tablet e0c1793b412c4608b1e004525a6dd887 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 20:14:54.862427 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412dab7986704f48be75b1fdbdccc9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.864883 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c1793b412c4608b1e004525a6dd887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.868255 30919 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:54.873741 30919 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.865247 30781 tablet_service.cc:1467] Processing CreateTablet for tablet b060012888ff4d76b254b43d1f1959bb (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 20:14:54.818606 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c1793b412c4608b1e004525a6dd887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.876536 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:54.867498 30858 tablet_service.cc:1467] Processing CreateTablet for tablet 9f32336173df4611829be37d396e86ba (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 20:14:54.870918 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba1969a3911423e8e51fc4f2962515d (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
W20260806 20:14:54.878016 30834 leader_election.cc:341] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:54.883596 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:54.884270 30791 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:54.885381 30835 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:54.890883 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1969a3911423e8e51fc4f2962515d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.892627 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b060012888ff4d76b254b43d1f1959bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.876426 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b060012888ff4d76b254b43d1f1959bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.892844 30699 tablet_service.cc:1467] Processing CreateTablet for tablet 2b0fcea699eb4ec884b7451428998012 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 20:14:54.896405 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c541d91c0c4487c9a76c6fc6f81a95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.876372 30779 tablet_service.cc:1467] Processing CreateTablet for tablet 15c9bdefd8144ce787ecb60a54ea1450 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 20:14:54.897388 30850 tablet_service.cc:1467] Processing CreateTablet for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 20:14:54.922194 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.922391 30773 tablet_service.cc:1467] Processing CreateTablet for tablet 2b0fcea699eb4ec884b7451428998012 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 20:14:54.923553 30773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0fcea699eb4ec884b7451428998012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.898941 30693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c9bdefd8144ce787ecb60a54ea1450. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.899461 30914 tablet_bootstrap.cc:492] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:54.866014 30697 tablet_service.cc:1467] Processing CreateTablet for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 20:14:54.927321 30697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.904052 30919 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:54.895364 30859 tablet_service.cc:1467] Processing CreateTablet for tablet 412dab7986704f48be75b1fdbdccc9fa (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 20:14:54.915418 30782 tablet_service.cc:1467] Processing CreateTablet for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 20:14:54.931411 30852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b060012888ff4d76b254b43d1f1959bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.933738 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412dab7986704f48be75b1fdbdccc9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.934055 30919 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:54.934434 30919 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:54.934767 30919 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:54.938133 30858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f32336173df4611829be37d396e86ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.942600 30696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412dab7986704f48be75b1fdbdccc9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.931349 30779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c9bdefd8144ce787ecb60a54ea1450. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.938056 30694 tablet_service.cc:1467] Processing CreateTablet for tablet cca4a581dd9f4cff973793835675b6c3 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 20:14:54.945304 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.932966 30698 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee7927d05584c999f6e0a2b753336f1 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 20:14:54.945765 30914 tablet_bootstrap.cc:654] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.917496 30778 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba1969a3911423e8e51fc4f2962515d (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 20:14:54.919508 30913 ts_tablet_manager.cc:1428] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.069s user 0.008s sys 0.000s
I20260806 20:14:54.951970 30775 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee7927d05584c999f6e0a2b753336f1 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 20:14:54.957731 30702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f32336173df4611829be37d396e86ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.971223 30913 tablet_bootstrap.cc:492] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:54.974143 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1969a3911423e8e51fc4f2962515d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.976037 30698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee7927d05584c999f6e0a2b753336f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.977516 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca4a581dd9f4cff973793835675b6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.977901 30856 tablet_service.cc:1467] Processing CreateTablet for tablet e0c1793b412c4608b1e004525a6dd887 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 20:14:54.979167 30856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c1793b412c4608b1e004525a6dd887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.984335 30913 tablet_bootstrap.cc:654] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.985731 30699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0fcea699eb4ec884b7451428998012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.986150 30775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee7927d05584c999f6e0a2b753336f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:54.987936 30919 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:54.996064 30915 tablet_bootstrap.cc:654] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:54.999030 30921 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:54.999533 30921 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.001605 30921 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.003517 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.004248 30791 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:55.005230 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:14:55.006623 30676 leader_election.cc:341] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.007354 30676 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:55.007719 30853 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba1969a3911423e8e51fc4f2962515d (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 20:14:54.992146 30855 tablet_service.cc:1467] Processing CreateTablet for tablet 15c9bdefd8144ce787ecb60a54ea1450 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 20:14:55.013347 30853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1969a3911423e8e51fc4f2962515d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:55.020907 30700 tablet_service.cc:1467] Processing CreateTablet for tablet 4430b33a44474cbfbc682c4bef43e2f5 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 20:14:55.022303 30700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4430b33a44474cbfbc682c4bef43e2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:55.009593 30921 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.023147 30921 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.023518 30921 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.024149 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:14:55.025539 30834 leader_election.cc:341] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.049335 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:55.050567 30855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c9bdefd8144ce787ecb60a54ea1450. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:55.055603 30851 tablet_service.cc:1467] Processing CreateTablet for tablet 2b0fcea699eb4ec884b7451428998012 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 20:14:55.056870 30851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b0fcea699eb4ec884b7451428998012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:55.052467 30791 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.057440 30692 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59110 (request call id 47) took 1112 ms. Trace:
I20260806 20:14:55.060154 30850 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee7927d05584c999f6e0a2b753336f1 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 20:14:55.061498 30850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee7927d05584c999f6e0a2b753336f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:55.062229 30695 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59110 (request call id 48) took 1070 ms. Trace:
I20260806 20:14:55.062594 30695 rpcz_store.cc:276] 0806 20:14:53.992356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.766267 (+773911us) service_pool.cc:227] Handling call
0806 20:14:54.789625 (+ 23358us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:54.789830 (+ 205us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:54.952468 (+162638us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.061681 (+109213us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.062171 (+ 490us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3574}
I20260806 20:14:55.069089 30859 tablet_service.cc:1467] Processing CreateTablet for tablet cca4a581dd9f4cff973793835675b6c3 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 20:14:55.074572 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca4a581dd9f4cff973793835675b6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:55.077411 30860 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60882 (request call id 43) took 1021 ms. Trace:
I20260806 20:14:55.077831 30860 rpcz_store.cc:276] 0806 20:14:54.056353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.729717 (+673364us) service_pool.cc:227] Handling call
0806 20:14:54.796504 (+ 66787us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:54.796607 (+ 103us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:54.881159 (+ 84552us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.076899 (+195740us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.077353 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":488}
I20260806 20:14:55.077843 30781 tablet_service.cc:1467] Processing CreateTablet for tablet 4430b33a44474cbfbc682c4bef43e2f5 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 20:14:55.079924 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4430b33a44474cbfbc682c4bef43e2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:55.057901 30692 rpcz_store.cc:276] 0806 20:14:53.944857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.757874 (+813017us) service_pool.cc:227] Handling call
0806 20:14:54.759602 (+ 1728us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:54.759694 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:54.888956 (+129262us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.056941 (+167985us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.057384 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:14:55.086951 30774 tablet_service.cc:1467] Processing CreateTablet for tablet cca4a581dd9f4cff973793835675b6c3 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 20:14:55.088194 30774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca4a581dd9f4cff973793835675b6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:55.088347 30914 tablet_bootstrap.cc:492] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.088794 30914 ts_tablet_manager.cc:1397] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.190s user 0.007s sys 0.007s
I20260806 20:14:55.100037 30857 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60882 (request call id 44) took 1043 ms. Trace:
I20260806 20:14:55.100458 30857 rpcz_store.cc:276] 0806 20:14:54.056666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.770841 (+714175us) service_pool.cc:227] Handling call
0806 20:14:54.801888 (+ 31047us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:54.802002 (+ 114us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:54.914267 (+112265us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.044642 (+130375us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.099975 (+ 55333us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1754}
I20260806 20:14:55.079741 30860 tablet_service.cc:1467] Processing CreateTablet for tablet 4430b33a44474cbfbc682c4bef43e2f5 (DEFAULT_TABLE table=test-workload [id=9525a22bcf824745aa69b29814e684c4]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 20:14:55.046780 30919 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:55.102456 30860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4430b33a44474cbfbc682c4bef43e2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:14:55.114741 30852 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60882 (request call id 48) took 1033 ms. Trace:
I20260806 20:14:55.114763 30791 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
W20260806 20:14:55.115731 30655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:14:55.115258 30852 rpcz_store.cc:276] 0806 20:14:54.081830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.826438 (+744608us) service_pool.cc:227] Handling call
0806 20:14:54.898007 (+ 71569us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:54.898216 (+ 209us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:55.107348 (+209132us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.114245 (+ 6897us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.114676 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":317312}
I20260806 20:14:55.116200 30835 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:55.117057 30919 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.117548 30919 raft_consensus.cc:686] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:55.118348 30919 consensus_queue.cc:237] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.119599 30914 raft_consensus.cc:348] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.120312 30914 raft_consensus.cc:374] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.120568 30914 raft_consensus.cc:729] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.122886 30915 tablet_bootstrap.cc:492] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.123358 30915 ts_tablet_manager.cc:1397] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.288s user 0.027s sys 0.000s
I20260806 20:14:55.126350 30913 tablet_bootstrap.cc:492] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.126832 30913 ts_tablet_manager.cc:1397] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.156s user 0.012s sys 0.004s
I20260806 20:14:55.128206 30915 raft_consensus.cc:348] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.128973 30915 raft_consensus.cc:374] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.129247 30915 raft_consensus.cc:729] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.129956 30915 consensus_queue.cc:260] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.132212 30856 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60882 (request call id 51) took 1003 ms. Trace:
I20260806 20:14:55.132616 30856 rpcz_store.cc:276] 0806 20:14:54.128341 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.976966 (+848625us) service_pool.cc:227] Handling call
0806 20:14:54.978716 (+ 1750us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:54.978817 (+ 101us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:55.048577 (+ 69760us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.131657 (+ 83080us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.132139 (+ 482us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:14:55.133785 30921 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.135169 30699 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59110 (request call id 55) took 1012 ms. Trace:
I20260806 20:14:55.135550 30699 rpcz_store.cc:276] 0806 20:14:54.123285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.891944 (+768659us) service_pool.cc:227] Handling call
0806 20:14:54.948086 (+ 56142us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:54.949092 (+ 1006us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:55.028803 (+ 79711us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.134532 (+105729us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.135111 (+ 579us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":93}
I20260806 20:14:55.135288 30702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59110 (request call id 49) took 1126 ms. Trace:
I20260806 20:14:55.136401 30702 rpcz_store.cc:276] 0806 20:14:54.008622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.766685 (+758063us) service_pool.cc:227] Handling call
0806 20:14:54.790250 (+ 23565us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:54.790342 (+ 92us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:55.051468 (+261126us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.134841 (+ 83373us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.135232 (+ 391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:14:55.139806 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:55.140481 30791 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.142603 30696 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59110 (request call id 50) took 1125 ms. Trace:
I20260806 20:14:55.142966 30696 rpcz_store.cc:276] 0806 20:14:54.017557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.784049 (+766492us) service_pool.cc:227] Handling call
0806 20:14:54.816586 (+ 32537us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:54.816684 (+ 98us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:55.072587 (+255903us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.142045 (+ 69458us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.142534 (+ 489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:14:55.148052 30791 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:55.147979 30602 catalog_manager.cc:5495] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:55.149390 30676 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:55.155141 30921 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.155727 30921 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.156193 30921 raft_consensus.cc:686] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:55.121248 30914 consensus_queue.cc:260] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.160238 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:55.160406 30914 ts_tablet_manager.cc:1428] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.071s user 0.003s sys 0.004s
W20260806 20:14:55.164305 30676 leader_election.cc:341] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.165505 30913 raft_consensus.cc:348] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.166385 30913 raft_consensus.cc:374] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.166683 30913 raft_consensus.cc:729] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.167465 30913 consensus_queue.cc:260] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.157094 30921 consensus_queue.cc:237] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.171478 30915 ts_tablet_manager.cc:1428] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.048s user 0.020s sys 0.026s
I20260806 20:14:55.171959 30914 tablet_bootstrap.cc:492] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.172497 30915 tablet_bootstrap.cc:492] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:55.174064 30913 ts_tablet_manager.cc:1428] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.047s user 0.008s sys 0.000s
I20260806 20:14:55.178891 30915 tablet_bootstrap.cc:654] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.181880 30913 tablet_bootstrap.cc:492] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.179147 30914 tablet_bootstrap.cc:654] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.179797 30855 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:60882 (request call id 52) took 1051 ms. Trace:
I20260806 20:14:55.184108 30855 rpcz_store.cc:276] 0806 20:14:54.128654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:14:54.991187 (+862533us) service_pool.cc:227] Handling call
0806 20:14:55.033372 (+ 42185us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0806 20:14:55.038038 (+ 4666us) ts_tablet_manager.cc:676] Creating new metadata...
0806 20:14:55.083918 (+ 45880us) tablet_metadata.cc:641] Metadata flushed
0806 20:14:55.179409 (+ 95491us) tablet_replica.cc:205] Creating consensus instance
0806 20:14:55.179753 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:14:55.190369 30913 tablet_bootstrap.cc:654] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.194285 30915 tablet_bootstrap.cc:492] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.194749 30915 ts_tablet_manager.cc:1397] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.000s
I20260806 20:14:55.197837 30915 raft_consensus.cc:348] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.198505 30913 tablet_bootstrap.cc:492] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.198614 30915 raft_consensus.cc:374] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.198210 30602 catalog_manager.cc:5495] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:55.199126 30915 raft_consensus.cc:729] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.199366 30913 ts_tablet_manager.cc:1397] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.001s
I20260806 20:14:55.200069 30914 tablet_bootstrap.cc:492] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.199836 30915 consensus_queue.cc:260] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.200526 30914 ts_tablet_manager.cc:1397] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.003s
I20260806 20:14:55.202308 30915 ts_tablet_manager.cc:1428] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:14:55.202772 30913 raft_consensus.cc:348] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.203490 30915 tablet_bootstrap.cc:492] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:55.203186 30914 raft_consensus.cc:348] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.203601 30913 raft_consensus.cc:374] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.203989 30914 raft_consensus.cc:374] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.204224 30913 raft_consensus.cc:729] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.204288 30914 raft_consensus.cc:729] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.205248 30913 consensus_queue.cc:260] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.205307 30914 consensus_queue.cc:260] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.207985 30914 ts_tablet_manager.cc:1428] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:14:55.210106 30913 ts_tablet_manager.cc:1428] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260806 20:14:55.211676 30913 tablet_bootstrap.cc:492] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.213403 30915 tablet_bootstrap.cc:654] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.212229 30914 tablet_bootstrap.cc:492] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.219637 30914 tablet_bootstrap.cc:654] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.219712 30921 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.225991 30921 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.228127 30921 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.235838 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.235837 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.237332 30915 tablet_bootstrap.cc:492] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.237423 30871 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:55.237898 30915 ts_tablet_manager.cc:1397] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.014s
I20260806 20:14:55.238452 30914 tablet_bootstrap.cc:492] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.238951 30914 ts_tablet_manager.cc:1397] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.005s
W20260806 20:14:55.238914 30676 leader_election.cc:341] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.239801 30676 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:55.240908 30921 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.241850 30921 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.242202 30921 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.242719 30913 tablet_bootstrap.cc:654] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.242048 30914 raft_consensus.cc:348] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.243115 30914 raft_consensus.cc:374] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.243455 30914 raft_consensus.cc:729] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.244319 30914 consensus_queue.cc:260] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.249821 30921 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.251510 30914 ts_tablet_manager.cc:1428] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260806 20:14:55.252475 30914 tablet_bootstrap.cc:492] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.252797 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:14:55.254292 30676 leader_election.cc:341] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.240602 30915 raft_consensus.cc:348] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.254907 30915 raft_consensus.cc:374] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.255218 30915 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.256101 30915 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.258592 30914 tablet_bootstrap.cc:654] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.262181 30943 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.262594 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:55.263249 30871 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.262681 30943 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.264367 30913 tablet_bootstrap.cc:492] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.264834 30913 ts_tablet_manager.cc:1397] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.053s user 0.025s sys 0.001s
I20260806 20:14:55.265337 30943 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.265914 30921 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.267694 30915 ts_tablet_manager.cc:1428] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260806 20:14:55.267521 30913 raft_consensus.cc:348] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.268282 30913 raft_consensus.cc:374] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.268575 30913 raft_consensus.cc:729] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.269289 30913 consensus_queue.cc:260] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.268679 30915 tablet_bootstrap.cc:492] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:55.270563 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.271443 30913 ts_tablet_manager.cc:1428] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 20:14:55.272321 30913 tablet_bootstrap.cc:492] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.275205 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:14:55.276552 30676 leader_election.cc:341] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.277128 30915 tablet_bootstrap.cc:654] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
W20260806 20:14:55.277216 30676 leader_election.cc:341] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.277724 30676 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:55.278470 30921 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:55.276177 30871 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:55.280678 30676 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:55.281400 30921 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.281877 30921 raft_consensus.cc:686] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:55.282717 30921 consensus_queue.cc:237] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.284286 30915 tablet_bootstrap.cc:492] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.284775 30915 ts_tablet_manager.cc:1397] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260806 20:14:55.286285 30914 tablet_bootstrap.cc:492] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.286885 30914 ts_tablet_manager.cc:1397] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.005s
I20260806 20:14:55.287606 30915 raft_consensus.cc:348] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.288472 30913 tablet_bootstrap.cc:654] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.288328 30915 raft_consensus.cc:374] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.289420 30915 raft_consensus.cc:729] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.289964 30914 raft_consensus.cc:348] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.290302 30915 consensus_queue.cc:260] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.290853 30914 raft_consensus.cc:374] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.291227 30914 raft_consensus.cc:729] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.292126 30914 consensus_queue.cc:260] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.292788 30915 ts_tablet_manager.cc:1428] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260806 20:14:55.296208 30915 tablet_bootstrap.cc:492] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:55.299136 30914 ts_tablet_manager.cc:1428] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260806 20:14:55.300325 30914 tablet_bootstrap.cc:492] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.301151 30603 catalog_manager.cc:5495] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:55.302983 30915 tablet_bootstrap.cc:654] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.311149 30914 tablet_bootstrap.cc:654] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.319097 30915 tablet_bootstrap.cc:492] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.319187 30913 tablet_bootstrap.cc:492] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.318795 30914 tablet_bootstrap.cc:492] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.319763 30915 ts_tablet_manager.cc:1397] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.013s
I20260806 20:14:55.319823 30913 ts_tablet_manager.cc:1397] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.018s
I20260806 20:14:55.319927 30914 ts_tablet_manager.cc:1397] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.006s
I20260806 20:14:55.321373 30922 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.322016 30922 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.323427 30914 raft_consensus.cc:348] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.324012 30914 raft_consensus.cc:374] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.323670 30913 raft_consensus.cc:348] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.324282 30914 raft_consensus.cc:729] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.325063 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:55.324957 30914 consensus_queue.cc:260] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.325716 30915 raft_consensus.cc:348] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:14:55.326467 30756 leader_election.cc:341] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.326673 30913 raft_consensus.cc:374] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.327011 30914 ts_tablet_manager.cc:1428] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260806 20:14:55.326421 30915 raft_consensus.cc:374] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.325208 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.327111 30913 raft_consensus.cc:729] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.327593 30915 raft_consensus.cc:729] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.327782 30871 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:55.324021 30922 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.328069 30913 consensus_queue.cc:260] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.328272 30915 consensus_queue.cc:260] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.329133 30757 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:55.330464 30915 ts_tablet_manager.cc:1428] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260806 20:14:55.331676 30915 tablet_bootstrap.cc:492] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:55.332372 30922 raft_consensus.cc:2781] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.332723 30922 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.333052 30922 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.333854 30913 ts_tablet_manager.cc:1428] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260806 20:14:55.334828 30913 tablet_bootstrap.cc:492] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.338223 30915 tablet_bootstrap.cc:654] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.340816 30913 tablet_bootstrap.cc:654] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.341503 30914 tablet_bootstrap.cc:492] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.341586 30921 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.342092 30921 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.344175 30921 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.346320 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.346550 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.347271 30871 raft_consensus.cc:2454] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
W20260806 20:14:55.347723 30676 leader_election.cc:341] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.348461 30676 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:55.349561 30921 raft_consensus.cc:2781] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.349895 30921 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.350208 30921 raft_consensus.cc:3037] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.353264 30915 tablet_bootstrap.cc:492] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.353864 30915 ts_tablet_manager.cc:1397] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.001s
I20260806 20:14:55.340382 30922 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.360046 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:55.360136 30922 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:14:55.361514 30756 leader_election.cc:341] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.362183 30913 tablet_bootstrap.cc:492] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.362622 30913 ts_tablet_manager.cc:1397] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.008s
I20260806 20:14:55.363034 30914 tablet_bootstrap.cc:654] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.364511 30913 raft_consensus.cc:348] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.365018 30913 raft_consensus.cc:374] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.365222 30913 raft_consensus.cc:729] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.356881 30915 raft_consensus.cc:348] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.365886 30915 raft_consensus.cc:374] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.365787 30913 consensus_queue.cc:260] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.366721 30921 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.367614 30913 ts_tablet_manager.cc:1428] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 20:14:55.368296 30913 tablet_bootstrap.cc:492] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.368754 30921 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.366238 30915 raft_consensus.cc:729] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.369701 30915 consensus_queue.cc:260] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.369768 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:55.371932 30915 ts_tablet_manager.cc:1428] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260806 20:14:55.372536 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:55.372828 30915 tablet_bootstrap.cc:492] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:55.373220 30871 raft_consensus.cc:3037] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.373431 30913 tablet_bootstrap.cc:654] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
W20260806 20:14:55.374936 30676 leader_election.cc:341] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.378737 30915 tablet_bootstrap.cc:654] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.382843 30871 raft_consensus.cc:2454] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:55.384701 30676 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:55.385756 30921 raft_consensus.cc:2781] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.386299 30921 raft_consensus.cc:686] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:55.387326 30921 consensus_queue.cc:237] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:14:55.390713 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 61665.1us
I20260806 20:14:55.394161 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:55.394891 30870 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.395772 30914 tablet_bootstrap.cc:492] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.396303 30914 ts_tablet_manager.cc:1397] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.055s user 0.011s sys 0.000s
I20260806 20:14:55.398034 30943 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.398629 30943 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.399864 30914 raft_consensus.cc:348] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.400724 30914 raft_consensus.cc:374] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.400815 30943 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.401069 30914 raft_consensus.cc:729] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.402175 30914 consensus_queue.cc:260] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.406088 30914 ts_tablet_manager.cc:1428] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260806 20:14:55.407665 30914 tablet_bootstrap.cc:492] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.416887 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.417779 30791 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:55.420006 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.420877 30871 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:55.420017 30676 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:55.423166 30943 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.423570 30943 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.423889 30943 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.425853 30914 tablet_bootstrap.cc:654] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.429672 30913 tablet_bootstrap.cc:492] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.430174 30913 ts_tablet_manager.cc:1397] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.062s user 0.041s sys 0.020s
I20260806 20:14:55.432557 30870 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:55.432842 30913 raft_consensus.cc:348] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.433784 30913 raft_consensus.cc:374] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.434105 30913 raft_consensus.cc:729] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.434837 30913 consensus_queue.cc:260] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.435869 30914 tablet_bootstrap.cc:492] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.435384 30943 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.436329 30914 ts_tablet_manager.cc:1397] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s
I20260806 20:14:55.437237 30913 ts_tablet_manager.cc:1428] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:14:55.438261 30913 tablet_bootstrap.cc:492] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.438550 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:55.438737 30914 raft_consensus.cc:348] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.439357 30791 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.445077 30913 tablet_bootstrap.cc:654] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.447589 30757 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:55.448752 30922 raft_consensus.cc:2781] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.449263 30922 raft_consensus.cc:686] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:55.450153 30922 consensus_queue.cc:237] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.454990 30791 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:55.456228 30676 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:55.458312 30921 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.458806 30921 raft_consensus.cc:686] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:55.460635 30921 consensus_queue.cc:237] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.461850 30603 catalog_manager.cc:5495] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:55.474427 30943 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.480058 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:55.480593 30915 tablet_bootstrap.cc:492] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.480700 30870 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.481074 30915 ts_tablet_manager.cc:1397] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.108s user 0.029s sys 0.062s
I20260806 20:14:55.481606 30913 tablet_bootstrap.cc:492] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.482029 30913 ts_tablet_manager.cc:1397] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.015s
I20260806 20:14:55.482981 30603 catalog_manager.cc:5495] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:55.484275 30913 raft_consensus.cc:348] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.484941 30913 raft_consensus.cc:374] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.485206 30913 raft_consensus.cc:729] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.483520 30915 raft_consensus.cc:348] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.486285 30915 raft_consensus.cc:374] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.485929 30913 consensus_queue.cc:260] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.486717 30915 raft_consensus.cc:729] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.487555 30915 consensus_queue.cc:260] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.488301 30913 ts_tablet_manager.cc:1428] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:14:55.489193 30913 tablet_bootstrap.cc:492] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.489938 30915 ts_tablet_manager.cc:1428] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260806 20:14:55.490931 30915 tablet_bootstrap.cc:492] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:55.439431 30914 raft_consensus.cc:374] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.494814 30914 raft_consensus.cc:729] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.496735 30915 tablet_bootstrap.cc:654] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.497934 30914 consensus_queue.cc:260] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.500008 30914 ts_tablet_manager.cc:1428] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.063s user 0.007s sys 0.000s
I20260806 20:14:55.502874 30870 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:55.503321 30914 tablet_bootstrap.cc:492] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.502964 30602 catalog_manager.cc:5495] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:55.507321 30913 tablet_bootstrap.cc:654] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.509292 30914 tablet_bootstrap.cc:654] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.519850 30919 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.520398 30919 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.522719 30919 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:55.525516 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:55.526527 30913 tablet_bootstrap.cc:492] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.526301 30712 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:55.526976 30913 ts_tablet_manager.cc:1397] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.010s
I20260806 20:14:55.528944 30834 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:55.529814 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.530484 30919 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.530501 30791 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:55.530920 30919 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.531230 30919 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.535027 30913 raft_consensus.cc:348] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.535727 30913 raft_consensus.cc:374] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.536015 30913 raft_consensus.cc:729] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.536702 30913 consensus_queue.cc:260] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.537606 30919 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.538209 30915 tablet_bootstrap.cc:492] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.538728 30915 ts_tablet_manager.cc:1397] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.016s
I20260806 20:14:55.539268 30913 ts_tablet_manager.cc:1428] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260806 20:14:55.540385 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:55.540987 30712 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.541472 30913 tablet_bootstrap.cc:492] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.541198 30915 raft_consensus.cc:348] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.541931 30915 raft_consensus.cc:374] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.542179 30915 raft_consensus.cc:729] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.542784 30915 consensus_queue.cc:260] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.544922 30915 ts_tablet_manager.cc:1428] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:14:55.546070 30919 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:55.540611 30914 tablet_bootstrap.cc:492] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.546779 30914 ts_tablet_manager.cc:1397] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.011s
I20260806 20:14:55.547032 30913 tablet_bootstrap.cc:654] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.547509 30712 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:55.548638 30834 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:55.549216 30914 raft_consensus.cc:348] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.549625 30919 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.549971 30914 raft_consensus.cc:374] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.550125 30919 raft_consensus.cc:686] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:55.550294 30914 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.551002 30919 consensus_queue.cc:237] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.551757 30915 tablet_bootstrap.cc:492] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:55.551944 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:55.552518 30791 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.558081 30915 tablet_bootstrap.cc:654] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.560631 30937 consensus_queue.cc:1035] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:55.563973 30913 tablet_bootstrap.cc:492] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.564464 30913 ts_tablet_manager.cc:1397] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.003s
I20260806 20:14:55.567075 30913 raft_consensus.cc:348] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.551021 30914 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.567917 30913 raft_consensus.cc:374] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.568218 30913 raft_consensus.cc:729] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.568986 30913 consensus_queue.cc:260] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.571369 30913 ts_tablet_manager.cc:1428] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:14:55.571503 30914 ts_tablet_manager.cc:1428] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260806 20:14:55.572289 30913 tablet_bootstrap.cc:492] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.572400 30914 tablet_bootstrap.cc:492] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.578421 30913 tablet_bootstrap.cc:654] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.577378 30791 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:55.584365 30914 tablet_bootstrap.cc:654] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.601790 30921 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.602370 30921 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.604907 30921 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.606648 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.608228 30712 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.606648 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.610107 30676 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:55.614087 30937 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.615837 30943 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:55.614650 30937 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.619254 30937 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:55.620832 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:55.621536 30713 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:55.622706 30834 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:55.624522 30937 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.624958 30937 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.625355 30937 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.636832 30921 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:55.640404 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.643379 30791 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:55.659011 30990 consensus_queue.cc:1035] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:55.665700 30919 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.666260 30919 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.633001 30937 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.677963 30870 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.676705 30922 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.679286 30922 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.684381 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:55.683357 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.688337 30871 raft_consensus.cc:2379] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 1: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:14:55.691174 30757 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:55.692317 30965 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.681315 30922 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.701471 30914 tablet_bootstrap.cc:492] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.702497 30914 ts_tablet_manager.cc:1397] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.130s user 0.017s sys 0.025s
I20260806 20:14:55.704792 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:55.709407 30965 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:55.710392 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:55.710659 30937 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:55.710623 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.713042 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:55.713694 30791 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.715039 30915 tablet_bootstrap.cc:492] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.714221 30602 catalog_manager.cc:5495] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:55.715497 30915 ts_tablet_manager.cc:1397] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.164s user 0.038s sys 0.042s
I20260806 20:14:55.711020 30712 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.697494 30919 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:55.715844 30914 raft_consensus.cc:348] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.716562 30914 raft_consensus.cc:374] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.716813 30914 raft_consensus.cc:729] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.717378 30914 consensus_queue.cc:260] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.717881 30835 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:55.718763 30992 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:55.719570 30915 raft_consensus.cc:348] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.720294 30915 raft_consensus.cc:374] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.720537 30915 raft_consensus.cc:729] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.720695 30914 ts_tablet_manager.cc:1428] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260806 20:14:55.713311 30944 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260806 20:14:55.721726 30913 tablet_bootstrap.cc:492] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.721747 30914 tablet_bootstrap.cc:492] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.722271 30913 ts_tablet_manager.cc:1397] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.150s user 0.058s sys 0.050s
I20260806 20:14:55.724890 30915 consensus_queue.cc:260] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.730463 30915 ts_tablet_manager.cc:1428] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.015s user 0.006s sys 0.002s
I20260806 20:14:55.731498 30915 tablet_bootstrap.cc:492] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:55.803498 30922 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.804188 30922 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.806713 30922 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.808064 30791 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:55.809010 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:55.809356 30835 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:55.810318 30990 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.810812 30990 raft_consensus.cc:686] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:55.810969 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.811823 30990 consensus_queue.cc:237] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:14:55.812965 30757 leader_election.cc:341] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:14:55.814060 30756 leader_election.cc:341] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.814476 30756 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:55.814929 30921 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.815441 30921 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.815814 30943 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:55.817492 30944 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.817955 30944 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.820880 30922 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:55.822635 30921 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.825001 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.825244 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.825299 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.827744 30869 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:55.828375 30868 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:55.825891 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.823103 30944 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.826125 30943 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:55.829236 30790 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:55.829733 30676 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:55.828091 30789 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:55.831004 30913 raft_consensus.cc:348] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.831753 30913 raft_consensus.cc:374] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.832036 30913 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.832490 30676 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:55.832741 30913 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.833583 30921 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.834028 30921 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.834437 30921 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.834506 30943 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.836134 30943 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.836800 30943 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.838404 30712 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:55.844388 30915 tablet_bootstrap.cc:654] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.850031 30921 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.852182 30943 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.853035 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:55.853840 30792 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.861901 30914 tablet_bootstrap.cc:654] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.865590 30943 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.866210 30943 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.866406 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:55.867051 30789 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.868008 30913 ts_tablet_manager.cc:1428] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.145s user 0.006s sys 0.000s
I20260806 20:14:55.868992 30792 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:55.870684 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:55.871799 30943 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.873346 30676 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:55.873924 30943 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.868922 30913 tablet_bootstrap.cc:492] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:55.874255 30944 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.874806 30944 raft_consensus.cc:686] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:55.875838 30944 consensus_queue.cc:237] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.878821 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.878948 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.879618 30914 tablet_bootstrap.cc:492] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:55.880122 30914 ts_tablet_manager.cc:1397] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.159s user 0.008s sys 0.001s
I20260806 20:14:55.871343 30869 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.880465 30792 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:55.899499 30789 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:55.910209 30913 tablet_bootstrap.cc:654] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:55.915323 30914 raft_consensus.cc:348] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.916115 30914 raft_consensus.cc:374] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.916376 30914 raft_consensus.cc:729] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.918419 30914 consensus_queue.cc:260] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.932346 30914 ts_tablet_manager.cc:1428] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.052s user 0.005s sys 0.008s
I20260806 20:14:55.933796 30914 tablet_bootstrap.cc:492] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:55.944178 30991 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:55.944859 30991 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.955070 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:55.955216 30991 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:55.961884 30791 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
W20260806 20:14:55.929842 30676 leader_election.cc:341] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.972005 30869 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:55.972230 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:55.973906 30835 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:14:55.976322 30834 leader_election.cc:341] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:55.976920 30913 tablet_bootstrap.cc:492] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:55.977370 30913 ts_tablet_manager.cc:1397] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.109s user 0.014s sys 0.005s
I20260806 20:14:55.972265 30921 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.974802 30994 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:14:55.971720 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 43642.6us
I20260806 20:14:55.979046 30994 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.981429 30676 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:55.982178 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:55.982618 30676 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:55.989585 30994 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:55.984052 30921 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.989001 30913 raft_consensus.cc:348] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.990373 30915 tablet_bootstrap.cc:492] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:55.990430 30913 raft_consensus.cc:374] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.990872 30915 ts_tablet_manager.cc:1397] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.260s user 0.046s sys 0.050s
I20260806 20:14:55.990820 30913 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:55.982533 30791 raft_consensus.cc:3037] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.990145 30921 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.992398 30999 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:55.992545 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:55.993033 30999 raft_consensus.cc:686] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:55.993119 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:55.993677 30921 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.993948 30870 raft_consensus.cc:2379] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 1: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:14:55.993899 30999 consensus_queue.cc:237] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.995254 30757 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:55.996946 30944 consensus_queue.cc:1035] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:55.984844 30990 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:55.997812 30990 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:55.998169 30990 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.982751 30869 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:55.998792 30915 raft_consensus.cc:348] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:55.999467 30915 raft_consensus.cc:374] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:55.999720 30915 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.002800 30915 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.005671 30913 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.011648 30913 ts_tablet_manager.cc:1428] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20260806 20:14:56.015215 30913 tablet_bootstrap.cc:492] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:56.021546 30994 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:56.029048 30915 ts_tablet_manager.cc:1428] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.038s user 0.010s sys 0.018s
I20260806 20:14:56.030064 30915 tablet_bootstrap.cc:492] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:56.010980 30921 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.032150 30869 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:56.035651 30944 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.036931 30999 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260806 20:14:56.039368 30756 consensus_peers.cc:487] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 -> Peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Couldn't send request to peer 42cabca48db7403b8523a3382a046043. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:56.043210 30921 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.044170 30914 tablet_bootstrap.cc:654] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.045403 30994 consensus_queue.cc:1035] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:56.049957 30913 tablet_bootstrap.cc:654] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.045089 30601 catalog_manager.cc:5495] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:56.056026 30990 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.059839 30990 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.061002 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:56.061785 30792 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.061089 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:56.062855 30713 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.065358 30915 tablet_bootstrap.cc:654] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.068168 30991 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.069110 30991 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.072036 30991 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.072942 30792 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:56.073137 30991 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.073319 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:56.075964 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.077333 30835 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:14:56.078274 30835 leader_election.cc:341] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.076359 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:14:56.079389 30834 leader_election.cc:341] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.079704 30789 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.079846 30834 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:56.079331 30990 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:56.080709 30990 raft_consensus.cc:686] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:56.081873 30995 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:56.081831 30914 tablet_bootstrap.cc:492] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:56.076663 30991 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.083159 30990 consensus_queue.cc:237] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.088248 30991 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.088836 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.089655 30914 ts_tablet_manager.cc:1397] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.156s user 0.020s sys 0.009s
I20260806 20:14:56.090085 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:14:56.091746 30835 leader_election.cc:341] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:14:56.093658 30834 leader_election.cc:341] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.094105 30713 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:56.094172 30834 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:56.088792 30789 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:56.095956 30991 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:56.097867 30676 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:56.099400 30999 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:56.099506 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:56.099892 30999 raft_consensus.cc:686] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:56.100688 30999 consensus_queue.cc:237] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:14:56.102157 30676 leader_election.cc:341] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.105111 30914 raft_consensus.cc:348] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.104614 31009 consensus_queue.cc:1035] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260806 20:14:56.111550 30914 raft_consensus.cc:374] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.111917 30914 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.112715 30914 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.115221 30914 ts_tablet_manager.cc:1428] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260806 20:14:56.116221 30914 tablet_bootstrap.cc:492] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
W20260806 20:14:56.118575 30655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:14:56.124439 30915 tablet_bootstrap.cc:492] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:56.124949 30915 ts_tablet_manager.cc:1397] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.005s
I20260806 20:14:56.128391 30913 tablet_bootstrap.cc:492] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:56.128881 30913 ts_tablet_manager.cc:1397] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.114s user 0.015s sys 0.002s
I20260806 20:14:56.131389 30913 raft_consensus.cc:348] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.132100 30913 raft_consensus.cc:374] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.132400 30913 raft_consensus.cc:729] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.133215 30914 tablet_bootstrap.cc:654] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.134479 30913 consensus_queue.cc:260] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.137926 30913 ts_tablet_manager.cc:1428] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260806 20:14:56.138748 30913 tablet_bootstrap.cc:492] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:56.141502 30915 raft_consensus.cc:348] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.142292 30915 raft_consensus.cc:374] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.142555 30915 raft_consensus.cc:729] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.146199 30602 catalog_manager.cc:5495] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:56.168779 30915 consensus_queue.cc:260] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.171950 30990 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.174804 30915 ts_tablet_manager.cc:1428] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.050s user 0.006s sys 0.027s
I20260806 20:14:56.175796 30915 tablet_bootstrap.cc:492] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:56.178100 30914 tablet_bootstrap.cc:492] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:56.178710 30914 ts_tablet_manager.cc:1397] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.006s
I20260806 20:14:56.180238 30913 tablet_bootstrap.cc:654] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.181524 30603 catalog_manager.cc:5495] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:56.181937 30990 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:56.183121 30603 catalog_manager.cc:5495] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:56.185859 30603 catalog_manager.cc:5495] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:56.186772 30914 raft_consensus.cc:348] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.187616 30914 raft_consensus.cc:374] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.187917 30914 raft_consensus.cc:729] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.188685 30914 consensus_queue.cc:260] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.189139 30915 tablet_bootstrap.cc:654] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.191709 30914 ts_tablet_manager.cc:1428] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:14:56.192698 30914 tablet_bootstrap.cc:492] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:56.209524 30922 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.210216 30922 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.213035 30922 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.217408 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.217806 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:56.218411 30868 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
W20260806 20:14:56.218961 30756 leader_election.cc:341] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.228401 30757 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:56.232283 30922 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:56.232904 30922 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:56.233371 30922 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.234543 30919 consensus_queue.cc:1035] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.247334 30913 tablet_bootstrap.cc:492] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:56.247799 30913 ts_tablet_manager.cc:1397] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.109s user 0.015s sys 0.026s
I20260806 20:14:56.250666 30913 raft_consensus.cc:348] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.251359 30913 raft_consensus.cc:374] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.251614 30913 raft_consensus.cc:729] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.252307 30913 consensus_queue.cc:260] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.254655 30913 ts_tablet_manager.cc:1428] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260806 20:14:56.255697 30913 tablet_bootstrap.cc:492] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:56.256208 30995 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.256729 30995 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.262473 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.263116 30914 tablet_bootstrap.cc:654] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.261245 30995 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:14:56.264225 30834 leader_election.cc:341] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.267035 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:14:56.268327 30835 leader_election.cc:341] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.268904 30835 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:56.270411 30990 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:56.272552 30913 tablet_bootstrap.cc:654] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.279239 30915 tablet_bootstrap.cc:492] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:56.279949 30915 ts_tablet_manager.cc:1397] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.104s user 0.019s sys 0.007s
I20260806 20:14:56.286494 30965 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.286607 31009 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.289484 30922 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.287099 30965 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.292383 30922 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.295552 30965 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.295737 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:56.296339 30868 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.297259 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:56.297761 30919 consensus_queue.cc:1035] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.298012 30869 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:56.299382 30757 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:56.301947 30965 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:56.302322 30965 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:56.302667 30965 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.307884 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:56.309859 30915 raft_consensus.cc:348] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.310631 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.311307 30712 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
W20260806 20:14:56.311628 30756 leader_election.cc:341] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.310667 30915 raft_consensus.cc:374] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.314746 30914 tablet_bootstrap.cc:492] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:56.315191 30914 ts_tablet_manager.cc:1397] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.123s user 0.015s sys 0.006s
I20260806 20:14:56.312489 30915 raft_consensus.cc:729] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.318506 30915 consensus_queue.cc:260] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.318241 30914 raft_consensus.cc:348] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.319892 30914 raft_consensus.cc:374] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.320188 30914 raft_consensus.cc:729] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.320971 30914 consensus_queue.cc:260] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.327665 30868 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:56.329025 30757 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:56.330763 30922 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:56.331111 30914 ts_tablet_manager.cc:1428] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260806 20:14:56.331292 30922 raft_consensus.cc:686] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:56.332523 30915 ts_tablet_manager.cc:1428] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.045s user 0.003s sys 0.003s
I20260806 20:14:56.333675 30915 tablet_bootstrap.cc:492] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:56.335980 30914 tablet_bootstrap.cc:492] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:56.336057 30922 consensus_queue.cc:237] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.339459 30913 tablet_bootstrap.cc:492] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:56.339946 30913 ts_tablet_manager.cc:1397] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.084s user 0.008s sys 0.027s
I20260806 20:14:56.336987 30965 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.348917 30913 raft_consensus.cc:348] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.356217 30913 raft_consensus.cc:374] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.356511 30913 raft_consensus.cc:729] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.357322 30913 consensus_queue.cc:260] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.359948 30913 ts_tablet_manager.cc:1428] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260806 20:14:56.360939 30913 tablet_bootstrap.cc:492] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:56.366479 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:56.367244 30712 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.367336 30965 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.353020 30914 tablet_bootstrap.cc:654] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.355783 30915 tablet_bootstrap.cc:654] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.367257 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:56.370368 30868 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.374269 30944 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.379386 30712 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:56.381102 30756 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:56.381372 30868 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:56.384105 30965 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:56.386787 30965 raft_consensus.cc:686] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:56.388409 30913 tablet_bootstrap.cc:654] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.387948 30965 consensus_queue.cc:237] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.394915 30602 catalog_manager.cc:5495] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:56.405294 30914 tablet_bootstrap.cc:492] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:56.405808 30914 ts_tablet_manager.cc:1397] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.070s user 0.015s sys 0.006s
I20260806 20:14:56.407248 30915 tablet_bootstrap.cc:492] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:56.407788 30915 ts_tablet_manager.cc:1397] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.074s user 0.014s sys 0.000s
I20260806 20:14:56.408313 30914 raft_consensus.cc:348] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.409030 30914 raft_consensus.cc:374] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.410521 30914 raft_consensus.cc:729] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.411523 30915 raft_consensus.cc:348] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.412185 30915 raft_consensus.cc:374] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.412395 30915 raft_consensus.cc:729] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.412700 30913 tablet_bootstrap.cc:492] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:56.413208 30913 ts_tablet_manager.cc:1397] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.007s
I20260806 20:14:56.415925 30913 raft_consensus.cc:348] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.416604 30913 raft_consensus.cc:374] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.416882 30913 raft_consensus.cc:729] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.412925 30915 consensus_queue.cc:260] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.416229 30914 consensus_queue.cc:260] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.425645 30913 consensus_queue.cc:260] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.427785 30915 ts_tablet_manager.cc:1428] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.019s user 0.002s sys 0.012s
I20260806 20:14:56.428874 30915 tablet_bootstrap.cc:492] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:56.434669 30915 tablet_bootstrap.cc:654] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.436327 30914 ts_tablet_manager.cc:1428] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260806 20:14:56.438685 30913 ts_tablet_manager.cc:1428] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260806 20:14:56.439692 30913 tablet_bootstrap.cc:492] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:56.444568 30603 catalog_manager.cc:5495] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:56.445952 30915 tablet_bootstrap.cc:492] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:56.446499 30915 ts_tablet_manager.cc:1397] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20260806 20:14:56.446621 30913 tablet_bootstrap.cc:654] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.444607 30914 tablet_bootstrap.cc:492] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:56.449208 30915 raft_consensus.cc:348] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.450112 30915 raft_consensus.cc:374] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.450448 30915 raft_consensus.cc:729] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.451169 30915 consensus_queue.cc:260] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.454412 30915 ts_tablet_manager.cc:1428] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:14:56.455473 30915 tablet_bootstrap.cc:492] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:56.461393 30915 tablet_bootstrap.cc:654] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.464349 30914 tablet_bootstrap.cc:654] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.468685 30944 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.469484 30944 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.471828 30944 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.474064 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:56.474996 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:14:56.475934 30676 leader_election.cc:341] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:14:56.479061 30676 leader_election.cc:341] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.479504 30676 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:56.480700 30944 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:56.481235 30913 tablet_bootstrap.cc:492] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:56.481663 30913 ts_tablet_manager.cc:1397] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.018s
I20260806 20:14:56.482152 30914 tablet_bootstrap.cc:492] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:56.482152 30915 tablet_bootstrap.cc:492] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:56.482751 30915 ts_tablet_manager.cc:1397] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.005s
I20260806 20:14:56.482751 30914 ts_tablet_manager.cc:1397] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.000s
I20260806 20:14:56.483985 30913 raft_consensus.cc:348] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.484668 30913 raft_consensus.cc:374] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.484923 30913 raft_consensus.cc:729] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.485283 30915 raft_consensus.cc:348] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.486023 30915 raft_consensus.cc:374] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.485711 30944 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:56.485646 30913 consensus_queue.cc:260] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.486323 30915 raft_consensus.cc:729] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.485603 30914 raft_consensus.cc:348] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.486800 30914 raft_consensus.cc:374] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.487191 30914 raft_consensus.cc:729] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.487071 30915 consensus_queue.cc:260] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.488081 30914 consensus_queue.cc:260] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.490240 30914 ts_tablet_manager.cc:1428] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:14:56.490816 30915 ts_tablet_manager.cc:1428] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:14:56.491418 30914 tablet_bootstrap.cc:492] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:56.494264 30913 ts_tablet_manager.cc:1428] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260806 20:14:56.495967 30915 tablet_bootstrap.cc:492] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:56.498193 30914 tablet_bootstrap.cc:654] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.498232 30712 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.501842 30915 tablet_bootstrap.cc:654] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.504807 30913 tablet_bootstrap.cc:492] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:56.504566 30919 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.505352 30919 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.508010 30919 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.510742 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:56.511452 30792 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:56.512662 30835 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:56.514525 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.515200 30711 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:56.517025 30919 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:56.517403 30919 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:56.517766 30919 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.522424 30991 consensus_queue.cc:1035] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:56.545535 30944 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:14:56.545878 30914 tablet_bootstrap.cc:492] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:56.546069 30944 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.548668 30944 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.546443 30914 ts_tablet_manager.cc:1397] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.055s user 0.021s sys 0.015s
I20260806 20:14:56.552289 30914 raft_consensus.cc:348] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.553141 30914 raft_consensus.cc:374] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.554291 30991 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.553472 30914 raft_consensus.cc:729] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.554790 30991 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.555157 30914 consensus_queue.cc:260] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.557130 30991 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.557773 30914 ts_tablet_manager.cc:1428] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:14:56.558686 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.560094 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:56.560783 30792 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:56.562378 30914 tablet_bootstrap.cc:492] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:56.562234 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:14:56.564165 30834 leader_election.cc:341] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.565549 30835 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:56.567241 30919 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.573655 30913 tablet_bootstrap.cc:654] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.588999 30991 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:56.591405 30991 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:56.593477 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:56.594776 30713 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.596017 30914 tablet_bootstrap.cc:654] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.597939 30919 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.593611 30991 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.602030 30944 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.606007 30915 tablet_bootstrap.cc:492] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:56.606424 30915 ts_tablet_manager.cc:1397] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.111s user 0.012s sys 0.025s
I20260806 20:14:56.605208 31007 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:56.608214 30944 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.609081 30915 raft_consensus.cc:348] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.610095 30915 raft_consensus.cc:374] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.610297 30915 raft_consensus.cc:729] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.591825 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:56.614607 30944 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.615813 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:56.616494 30792 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:56.618453 30915 consensus_queue.cc:260] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.625397 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:56.626742 30792 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.626581 30676 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:56.636372 30991 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.638149 31007 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:14:56.633219 30915 ts_tablet_manager.cc:1428] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.027s user 0.003s sys 0.003s
I20260806 20:14:56.641672 30713 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:56.642969 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:56.643227 30834 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:56.638583 30676 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:56.644575 31007 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:56.644919 31007 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:56.645254 31007 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.638517 30991 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.656857 30792 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:56.657614 30919 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:56.661316 30919 raft_consensus.cc:686] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:56.662876 30913 tablet_bootstrap.cc:492] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:56.662319 30919 consensus_queue.cc:237] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.644943 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:56.663332 30913 ts_tablet_manager.cc:1397] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.168s user 0.024s sys 0.022s
W20260806 20:14:56.658736 30834 leader_election.cc:341] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.659961 30915 tablet_bootstrap.cc:492] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:56.670512 30913 raft_consensus.cc:348] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.671247 30913 raft_consensus.cc:374] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.671525 30913 raft_consensus.cc:729] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.671818 30990 consensus_queue.cc:1035] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.672138 30913 consensus_queue.cc:260] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.677594 30915 tablet_bootstrap.cc:654] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.679647 31007 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.682601 31007 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.684412 30913 ts_tablet_manager.cc:1428] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.021s user 0.005s sys 0.004s
I20260806 20:14:56.685935 30913 tablet_bootstrap.cc:492] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:56.690883 30990 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.691406 30990 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.693009 30914 tablet_bootstrap.cc:492] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:56.693548 30914 ts_tablet_manager.cc:1397] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.131s user 0.015s sys 0.034s
I20260806 20:14:56.694713 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.694852 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:56.695485 30792 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.697337 30990 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.698410 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:56.699338 30791 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 1: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:14:56.700562 30835 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:56.702577 30991 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:56.698386 30676 leader_election.cc:341] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.708865 30914 raft_consensus.cc:348] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.709705 30913 tablet_bootstrap.cc:654] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.709728 30914 raft_consensus.cc:374] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.710283 30914 raft_consensus.cc:729] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.710474 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:56.711063 30914 consensus_queue.cc:260] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.711093 30789 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.713342 30914 ts_tablet_manager.cc:1428] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260806 20:14:56.714305 30914 tablet_bootstrap.cc:492] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:56.719022 30789 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:56.720250 30914 tablet_bootstrap.cc:654] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.721284 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:56.725430 31007 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:56.727330 31007 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.727833 31007 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.720330 30600 catalog_manager.cc:5495] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:56.730047 31007 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.731254 30676 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:56.731901 31007 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:56.732609 31007 raft_consensus.cc:686] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:56.732658 30991 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:56.733588 31007 consensus_queue.cc:237] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.736465 30914 tablet_bootstrap.cc:492] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:56.736944 30914 ts_tablet_manager.cc:1397] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20260806 20:14:56.740275 30914 raft_consensus.cc:348] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.741007 30914 raft_consensus.cc:374] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.741283 30914 raft_consensus.cc:729] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.741968 30914 consensus_queue.cc:260] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.743242 30991 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.743680 30991 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.745648 30991 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.748804 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:14:56.750813 30834 leader_election.cc:341] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.752642 30914 ts_tablet_manager.cc:1428] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
W20260806 20:14:56.754863 30676 leader_election.cc:341] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.757165 30914 tablet_bootstrap.cc:492] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:56.758163 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:56.762126 30944 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.763778 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:56.764757 30870 raft_consensus.cc:2379] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 1: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:14:56.764907 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:14:56.767494 30835 leader_election.cc:341] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.767732 30792 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:56.768664 30965 consensus_queue.cc:1035] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.769541 30676 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:56.769547 30835 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:56.771034 30943 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.778932 30990 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:56.787523 30915 tablet_bootstrap.cc:492] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:56.787635 30914 tablet_bootstrap.cc:654] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.788031 30915 ts_tablet_manager.cc:1397] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.128s user 0.009s sys 0.001s
I20260806 20:14:56.788945 30835 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:56.790639 30990 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:56.791492 30990 raft_consensus.cc:686] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:56.792887 30913 tablet_bootstrap.cc:492] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:56.793352 30913 ts_tablet_manager.cc:1397] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.108s user 0.017s sys 0.010s
I20260806 20:14:56.792393 30990 consensus_queue.cc:237] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.794876 30965 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:56.795718 30913 raft_consensus.cc:348] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.796388 30913 raft_consensus.cc:374] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.796645 30913 raft_consensus.cc:729] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.797247 30913 consensus_queue.cc:260] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.799993 30913 ts_tablet_manager.cc:1428] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:14:56.816325 30943 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:56.820423 30603 catalog_manager.cc:5495] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:56.903421 30913 tablet_bootstrap.cc:492] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:56.907199 30915 raft_consensus.cc:348] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.907970 30915 raft_consensus.cc:374] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.909525 30915 raft_consensus.cc:729] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
W20260806 20:14:56.905705 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 50113.1us
I20260806 20:14:56.910328 30915 consensus_queue.cc:260] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.913991 30915 ts_tablet_manager.cc:1428] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.126s user 0.017s sys 0.041s
I20260806 20:14:56.915678 30915 tablet_bootstrap.cc:492] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:56.923659 30965 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260806 20:14:56.935762 30913 tablet_bootstrap.cc:654] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.948841 31066 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.949352 31066 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.951385 31066 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:56.946795 31065 consensus_queue.cc:1035] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260806 20:14:56.961586 30991 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.962196 30991 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.964387 30991 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.946023 30990 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.965322 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.965484 30990 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.966104 30712 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:56.967097 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.968909 30603 catalog_manager.cc:5495] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:56.976543 30834 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:56.976761 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:56.977169 30914 tablet_bootstrap.cc:492] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:56.977413 30791 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:56.977633 30914 ts_tablet_manager.cc:1397] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.221s user 0.014s sys 0.018s
I20260806 20:14:56.982445 30914 raft_consensus.cc:348] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.984268 30914 raft_consensus.cc:374] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:56.979635 30991 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:56.984509 30914 raft_consensus.cc:729] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:56.981266 31009 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:56.989575 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:56.990018 31009 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:14:56.990963 30835 leader_election.cc:341] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.986490 30990 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:56.986652 30991 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:14:56.979499 30756 leader_election.cc:341] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.994659 30915 tablet_bootstrap.cc:654] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:56.986159 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:56.992260 31009 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:14:56.999482 30834 leader_election.cc:341] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:56.987207 30914 consensus_queue.cc:260] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:56.998535 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:56.992244 30991 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:56.999945 30834 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:57.001958 30868 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:57.002306 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:57.002918 30990 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:57.002921 30792 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:57.005314 30914 ts_tablet_manager.cc:1428] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260806 20:14:57.006376 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:57.007068 30868 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:57.011739 30991 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.006319 30914 tablet_bootstrap.cc:492] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:57.011370 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:57.025051 30943 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:57.025516 30943 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:57.027976 30943 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.048032 30990 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.054311 30990 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.024643 30757 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:57.070685 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:57.071489 30713 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.076790 31065 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:57.077245 31065 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:57.077585 31065 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:57.087854 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 63321.7us
I20260806 20:14:57.115432 30919 consensus_queue.cc:1035] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:57.126456 30914 tablet_bootstrap.cc:654] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.138005 30713 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:57.139302 30834 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:57.140574 30995 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:57.141034 30943 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.176651 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:57.175935 30991 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:57.177363 30792 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:57.181337 30757 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.28.186.254:40443, user_credentials={real_user=slave}} blocked reactor thread for 91046.5us
I20260806 20:14:57.187971 30995 raft_consensus.cc:686] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:14:57.195499 30655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:14:57.198004 30943 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.197705 30792 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
W20260806 20:14:57.212399 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 71166.7us
I20260806 20:14:57.218845 30913 tablet_bootstrap.cc:492] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:57.219300 30913 ts_tablet_manager.cc:1397] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.316s user 0.040s sys 0.028s
I20260806 20:14:57.224279 30913 raft_consensus.cc:348] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.226542 30915 tablet_bootstrap.cc:492] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:57.227097 30915 ts_tablet_manager.cc:1397] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.312s user 0.030s sys 0.034s
I20260806 20:14:57.227702 30913 raft_consensus.cc:374] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.211977 30990 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:57.227981 30913 raft_consensus.cc:729] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.232395 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:57.227391 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:57.245756 30712 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:57.246208 30870 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.215301 30995 consensus_queue.cc:237] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.251574 30913 consensus_queue.cc:260] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.235579 30915 raft_consensus.cc:348] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.253482 30915 raft_consensus.cc:374] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.253767 30915 raft_consensus.cc:729] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.254806 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:57.255373 30792 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.255904 30914 tablet_bootstrap.cc:492] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:57.256345 30914 ts_tablet_manager.cc:1397] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.250s user 0.018s sys 0.004s
I20260806 20:14:57.257928 30915 consensus_queue.cc:260] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.256335 30943 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 20:14:57.260437 30914 raft_consensus.cc:348] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.261247 30914 raft_consensus.cc:374] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.261585 30914 raft_consensus.cc:729] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
W20260806 20:14:57.262344 30676 consensus_peers.cc:487] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 -> Peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Couldn't send request to peer 58d59beb597f4ed99d84d102ddb41ff5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:57.262253 30914 consensus_queue.cc:260] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.264120 31073 consensus_queue.cc:1035] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 20:14:57.266161 30915 ts_tablet_manager.cc:1428] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.039s user 0.009s sys 0.002s
I20260806 20:14:57.264827 30834 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:57.266525 30913 ts_tablet_manager.cc:1428] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.047s user 0.002s sys 0.007s
I20260806 20:14:57.267359 30915 tablet_bootstrap.cc:492] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:57.240263 31065 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.268069 30913 tablet_bootstrap.cc:492] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:57.257688 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:57.277241 30789 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:57.303188 30914 ts_tablet_manager.cc:1428] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.046s user 0.005s sys 0.001s
I20260806 20:14:57.316851 30914 tablet_bootstrap.cc:492] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:57.325034 31074 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:57.325559 31074 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:57.329301 30919 consensus_queue.cc:1035] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 20:14:57.331764 30913 tablet_bootstrap.cc:654] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.338882 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:57.339558 30713 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:57.341566 30834 consensus_peers.cc:487] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Couldn't send request to peer 42cabca48db7403b8523a3382a046043. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:57.346297 31065 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.346916 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:57.347481 30713 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:57.347584 30869 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.353307 30870 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:57.355168 30869 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:57.325914 31074 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.356317 30757 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:57.356505 30602 catalog_manager.cc:5495] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:57.360543 31065 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:57.361301 31065 raft_consensus.cc:686] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:57.362233 31065 consensus_queue.cc:237] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.366531 30792 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:57.365008 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:57.371680 30943 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:57.372627 30943 raft_consensus.cc:686] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:57.373534 30943 consensus_queue.cc:237] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.388092 30913 tablet_bootstrap.cc:492] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:57.388960 30913 ts_tablet_manager.cc:1397] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.121s user 0.006s sys 0.014s
I20260806 20:14:57.389580 30915 tablet_bootstrap.cc:654] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.404079 31074 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.406107 30914 tablet_bootstrap.cc:654] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.406371 30913 raft_consensus.cc:348] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.407172 30913 raft_consensus.cc:374] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.407473 30913 raft_consensus.cc:729] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.407299 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:57.408059 30711 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.408764 31074 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:57.408118 30913 consensus_queue.cc:260] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.415350 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:57.416082 30789 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.416953 30913 ts_tablet_manager.cc:1428] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.028s user 0.004s sys 0.003s
I20260806 20:14:57.417897 30913 tablet_bootstrap.cc:492] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:57.423108 30602 catalog_manager.cc:5495] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:57.424685 30789 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:57.426077 30835 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:57.427067 31074 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:57.427577 31074 raft_consensus.cc:686] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:57.428459 31074 consensus_queue.cc:237] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.429198 30913 tablet_bootstrap.cc:654] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.441902 30915 tablet_bootstrap.cc:492] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:57.442451 30915 ts_tablet_manager.cc:1397] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.176s user 0.018s sys 0.004s
I20260806 20:14:57.443921 30711 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:57.448101 30914 tablet_bootstrap.cc:492] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:57.449977 30914 ts_tablet_manager.cc:1397] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.133s user 0.015s sys 0.015s
I20260806 20:14:57.452682 30914 raft_consensus.cc:348] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.452333 30602 catalog_manager.cc:5495] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:57.453294 30914 raft_consensus.cc:374] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.453603 30914 raft_consensus.cc:729] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.454284 30914 consensus_queue.cc:260] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.456554 30914 ts_tablet_manager.cc:1428] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260806 20:14:57.457475 30914 tablet_bootstrap.cc:492] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:57.459554 30915 raft_consensus.cc:348] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.460292 30915 raft_consensus.cc:374] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.460563 30915 raft_consensus.cc:729] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.459424 31074 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.462098 31074 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.464213 31074 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:57.465389 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:57.465706 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:57.466526 30789 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
W20260806 20:14:57.466784 30834 leader_election.cc:341] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.467705 30835 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:57.469188 31074 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:57.469560 31074 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:57.469878 31074 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.472469 30914 tablet_bootstrap.cc:654] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.461205 30915 consensus_queue.cc:260] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.476835 30602 catalog_manager.cc:5495] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:57.479144 31074 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.483017 30915 ts_tablet_manager.cc:1428] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20260806 20:14:57.483968 30915 tablet_bootstrap.cc:492] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:57.487835 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:14:57.489208 30834 leader_election.cc:341] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.490039 30913 tablet_bootstrap.cc:492] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:57.490180 31074 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:57.490692 30913 ts_tablet_manager.cc:1397] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.073s user 0.011s sys 0.007s
I20260806 20:14:57.492010 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:57.492811 30789 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.493340 30913 raft_consensus.cc:348] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.494026 30913 raft_consensus.cc:374] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.494278 30913 raft_consensus.cc:729] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.494925 30913 consensus_queue.cc:260] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.497228 30913 ts_tablet_manager.cc:1428] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 20:14:57.498136 30913 tablet_bootstrap.cc:492] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:57.500985 30789 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:57.502368 30835 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:57.503571 31074 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:57.504923 31074 raft_consensus.cc:686] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:57.506561 30914 tablet_bootstrap.cc:492] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:57.507090 30914 ts_tablet_manager.cc:1397] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.004s
I20260806 20:14:57.506508 31074 consensus_queue.cc:237] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.508292 30913 tablet_bootstrap.cc:654] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.511801 30915 tablet_bootstrap.cc:654] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.509972 30914 raft_consensus.cc:348] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.513928 30914 raft_consensus.cc:374] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.514261 30914 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.514981 30914 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.517189 30914 ts_tablet_manager.cc:1428] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260806 20:14:57.518245 30914 tablet_bootstrap.cc:492] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:57.520167 30913 tablet_bootstrap.cc:492] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:57.520660 30913 ts_tablet_manager.cc:1397] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.002s
I20260806 20:14:57.523305 30913 raft_consensus.cc:348] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.524077 30913 raft_consensus.cc:374] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.524422 30913 raft_consensus.cc:729] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.525115 30913 consensus_queue.cc:260] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.526612 30914 tablet_bootstrap.cc:654] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.532882 30915 tablet_bootstrap.cc:492] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:57.533212 30913 ts_tablet_manager.cc:1428] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260806 20:14:57.533332 30915 ts_tablet_manager.cc:1397] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.013s
I20260806 20:14:57.534139 30913 tablet_bootstrap.cc:492] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:57.533946 30600 catalog_manager.cc:5495] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:57.535871 30915 raft_consensus.cc:348] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.536520 30915 raft_consensus.cc:374] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.536769 30915 raft_consensus.cc:729] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.539350 31074 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.539355 30915 consensus_queue.cc:260] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.539958 31074 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.542704 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:57.544803 31074 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:57.545185 30913 tablet_bootstrap.cc:654] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.547092 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:57.548151 30915 ts_tablet_manager.cc:1428] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20260806 20:14:57.549083 30915 tablet_bootstrap.cc:492] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:57.550036 30835 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:57.550837 31074 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:57.554430 30915 tablet_bootstrap.cc:654] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.562276 30915 tablet_bootstrap.cc:492] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:57.562724 30915 ts_tablet_manager.cc:1397] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260806 20:14:57.563553 30914 tablet_bootstrap.cc:492] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:57.564002 30914 ts_tablet_manager.cc:1397] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.019s
I20260806 20:14:57.565088 30915 raft_consensus.cc:348] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.568429 30915 raft_consensus.cc:374] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.568710 30915 raft_consensus.cc:729] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.567015 30914 raft_consensus.cc:348] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.569594 30914 raft_consensus.cc:374] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.569859 30914 raft_consensus.cc:729] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.570428 30914 consensus_queue.cc:260] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.572461 30914 ts_tablet_manager.cc:1428] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260806 20:14:57.572710 30913 tablet_bootstrap.cc:492] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:57.573158 30913 ts_tablet_manager.cc:1397] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.004s
I20260806 20:14:57.573310 30914 tablet_bootstrap.cc:492] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:57.569195 30915 consensus_queue.cc:260] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.576041 30913 raft_consensus.cc:348] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.577088 30913 raft_consensus.cc:374] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.577350 30913 raft_consensus.cc:729] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.578048 30915 ts_tablet_manager.cc:1428] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260806 20:14:57.577955 30913 consensus_queue.cc:260] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.579316 30915 tablet_bootstrap.cc:492] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:57.583310 30913 ts_tablet_manager.cc:1428] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260806 20:14:57.584287 30913 tablet_bootstrap.cc:492] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:57.584661 30914 tablet_bootstrap.cc:654] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.596061 30915 tablet_bootstrap.cc:654] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.603528 30913 tablet_bootstrap.cc:654] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.610566 31074 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.616603 31074 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.619159 31074 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:57.623238 30869 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.627957 31065 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.628547 31065 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.630682 31065 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.632267 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:57.632411 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:14:57.634395 30756 leader_election.cc:341] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.635761 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:14:57.637116 30834 leader_election.cc:341] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:14:57.644299 30835 leader_election.cc:341] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.645040 30835 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:57.646714 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:14:57.648891 30757 leader_election.cc:341] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.649317 30757 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:57.650138 31065 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:57.651320 30919 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:57.665026 30914 tablet_bootstrap.cc:492] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:57.665416 30915 tablet_bootstrap.cc:492] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:57.665586 30914 ts_tablet_manager.cc:1397] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.092s user 0.025s sys 0.038s
I20260806 20:14:57.665890 30915 ts_tablet_manager.cc:1397] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.087s user 0.040s sys 0.024s
I20260806 20:14:57.668507 30915 raft_consensus.cc:348] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.668895 30914 raft_consensus.cc:348] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.669631 30914 raft_consensus.cc:374] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.669898 30914 raft_consensus.cc:729] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.667819 30913 tablet_bootstrap.cc:492] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:57.670703 30914 consensus_queue.cc:260] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.671959 30913 ts_tablet_manager.cc:1397] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.005s
I20260806 20:14:57.673002 30914 ts_tablet_manager.cc:1428] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260806 20:14:57.669288 30915 raft_consensus.cc:374] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.673341 30915 raft_consensus.cc:729] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.674244 30914 tablet_bootstrap.cc:492] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:57.674026 30915 consensus_queue.cc:260] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.676028 30915 ts_tablet_manager.cc:1428] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260806 20:14:57.677729 30913 raft_consensus.cc:348] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.678426 30913 raft_consensus.cc:374] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.678680 30913 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.679268 30913 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.680411 30914 tablet_bootstrap.cc:654] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.681358 30913 ts_tablet_manager.cc:1428] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260806 20:14:57.681784 30915 tablet_bootstrap.cc:492] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:57.686237 30914 tablet_bootstrap.cc:492] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:57.686708 30914 ts_tablet_manager.cc:1397] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260806 20:14:57.688474 30915 tablet_bootstrap.cc:654] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.689795 30914 raft_consensus.cc:348] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.689363 30913 tablet_bootstrap.cc:492] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:57.690966 30914 raft_consensus.cc:374] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.691262 30914 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.691855 30914 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.693701 30914 ts_tablet_manager.cc:1428] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:14:57.694607 30914 tablet_bootstrap.cc:492] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:57.698585 30913 tablet_bootstrap.cc:654] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.699940 30914 tablet_bootstrap.cc:654] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.701483 30915 tablet_bootstrap.cc:492] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:57.702064 30915 ts_tablet_manager.cc:1397] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.000s
I20260806 20:14:57.705003 30915 raft_consensus.cc:348] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.705721 30915 raft_consensus.cc:374] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.705976 30915 raft_consensus.cc:729] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.706646 31074 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:57.706579 30915 consensus_queue.cc:260] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.712621 30915 ts_tablet_manager.cc:1428] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260806 20:14:57.713586 30915 tablet_bootstrap.cc:492] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:57.721732 30915 tablet_bootstrap.cc:654] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.733867 30915 tablet_bootstrap.cc:492] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:57.734359 30915 ts_tablet_manager.cc:1397] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.001s
I20260806 20:14:57.736934 30915 raft_consensus.cc:348] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.737668 30915 raft_consensus.cc:374] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.737937 30915 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.738528 30915 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.740535 30915 ts_tablet_manager.cc:1428] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:14:57.741847 30915 tablet_bootstrap.cc:492] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:57.744455 31065 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.745043 31065 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.747393 31065 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.748252 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:57.748979 30713 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:57.750509 30756 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:57.752525 31065 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:57.752985 31065 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:57.753437 31065 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.754628 30915 tablet_bootstrap.cc:654] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.765717 30999 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.766417 30999 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.768644 30999 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.769381 31066 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.769886 31066 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.770900 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:57.771266 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:57.771840 31066 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.771862 30787 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
W20260806 20:14:57.772589 30676 leader_election.cc:341] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.774003 30676 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:57.774755 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:57.775393 30712 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:57.779155 30756 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:57.780120 31066 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:57.780511 31066 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:57.780843 31066 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.785707 30999 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:57.786250 30999 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:57.786607 30999 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.794771 31074 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.795584 31074 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.799700 31088 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.800246 31088 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.805883 31088 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.806082 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:57.807384 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:57.808317 30791 raft_consensus.cc:2379] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 1: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:14:57.800626 31065 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.810771 30676 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:57.811686 31088 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.819046 31088 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:57.829779 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:57.829859 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:57.830204 30913 tablet_bootstrap.cc:492] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:57.830559 30869 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:57.830879 30913 ts_tablet_manager.cc:1397] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.142s user 0.012s sys 0.005s
I20260806 20:14:57.836563 30915 tablet_bootstrap.cc:492] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:57.837060 30915 ts_tablet_manager.cc:1397] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.021s
I20260806 20:14:57.838938 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:57.839664 30711 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.839509 30915 raft_consensus.cc:348] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.840214 30915 raft_consensus.cc:374] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.840471 30915 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.841109 30915 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.842195 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:57.843273 30915 ts_tablet_manager.cc:1428] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 20:14:57.843360 30999 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.854645 30713 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.855474 30999 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.824513 31066 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.836295 30913 raft_consensus.cc:348] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.856305 30913 raft_consensus.cc:374] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.856590 30913 raft_consensus.cc:729] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.857208 30913 consensus_queue.cc:260] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.857678 31066 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.858448 31065 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.858916 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:57.861639 30999 consensus_queue.cc:1035] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:57.845239 30990 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 20:14:57.845775 30834 leader_election.cc:341] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.846690 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:57.862994 30787 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:57.852205 30914 tablet_bootstrap.cc:492] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:57.854837 30915 tablet_bootstrap.cc:492] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:57.864339 30914 ts_tablet_manager.cc:1397] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.170s user 0.050s sys 0.054s
I20260806 20:14:57.867018 30914 raft_consensus.cc:348] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.867794 30914 raft_consensus.cc:374] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.868057 30914 raft_consensus.cc:729] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.868734 30914 consensus_queue.cc:260] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.872267 30915 tablet_bootstrap.cc:654] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.863837 30913 ts_tablet_manager.cc:1428] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260806 20:14:57.875779 30913 tablet_bootstrap.cc:492] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:57.875626 30835 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:57.876585 30995 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:57.876921 30995 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:57.877224 30995 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.877758 30914 ts_tablet_manager.cc:1428] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260806 20:14:57.861568 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:57.880035 30792 raft_consensus.cc:2379] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 1: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:14:57.871857 30711 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:57.884451 30913 tablet_bootstrap.cc:654] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.887534 30995 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:14:57.888652 30757 leader_election.cc:341] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.890174 30914 tablet_bootstrap.cc:492] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:57.890494 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:57.891645 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:57.892169 30871 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:57.895447 30676 leader_election.cc:341] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.844493 31074 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:14:57.897364 30757 leader_election.cc:341] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.897352 31112 consensus_queue.cc:1035] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:57.898149 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:57.898924 30711 raft_consensus.cc:2379] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 1: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:14:57.899507 30995 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:57.902772 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:57.903441 30711 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.904112 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:57.904740 30789 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.909647 30915 tablet_bootstrap.cc:492] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:57.909722 31112 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.910514 30915 ts_tablet_manager.cc:1397] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.003s
I20260806 20:14:57.910288 31112 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.899930 30756 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:57.915024 31065 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:57.915516 31065 raft_consensus.cc:686] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:57.915422 30676 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:57.916664 30756 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:57.916787 31065 consensus_queue.cc:237] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.917708 31074 consensus_queue.cc:1035] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:57.915742 31112 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.918340 30999 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:14:57.924276 31066 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 20:14:57.924715 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:57.924276 30871 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
W20260806 20:14:57.933485 30756 leader_election.cc:341] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.934201 30914 tablet_bootstrap.cc:654] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:57.934463 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:57.936359 30869 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:57.931632 31112 consensus_queue.cc:1035] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260806 20:14:57.937201 30789 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:57.938757 30757 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:57.939576 31066 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:57.939941 31066 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:57.940260 31066 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:57.941573 30999 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:57.942092 30999 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.955463 30835 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:57.956401 30995 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:57.960368 30915 raft_consensus.cc:348] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.960924 30915 raft_consensus.cc:374] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:57.961123 30915 raft_consensus.cc:729] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:57.961786 30915 consensus_queue.cc:260] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.962992 30999 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:57.963455 31066 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.965883 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:57.957191 30995 raft_consensus.cc:686] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:14:57.967422 30676 leader_election.cc:341] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:57.968123 30995 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:57.968492 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:57.969101 30870 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:57.970922 30676 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:57.972155 30711 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:57.972456 30943 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.010468 30915 ts_tablet_manager.cc:1428] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.100s user 0.006s sys 0.001s
I20260806 20:14:57.973289 30943 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.018675 30915 tablet_bootstrap.cc:492] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:58.012218 31088 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:58.011643 31066 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.032275 30991 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.032871 30991 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.034516 30990 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 20:14:58.015025 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 44199.5us
I20260806 20:14:58.035049 30991 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:58.027482 31088 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:58.095947 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:58.096657 30712 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.072537 31088 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.104009 30943 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.088927 31009 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:14:58.151219 30914 tablet_bootstrap.cc:492] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:58.152143 30914 ts_tablet_manager.cc:1397] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.262s user 0.016s sys 0.019s
I20260806 20:14:58.115622 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.155194 30712 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:58.142328 30990 consensus_queue.cc:1035] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260806 20:14:58.146456 31009 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.124923 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.142477 30995 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260806 20:14:58.155870 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:58.168776 30871 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:58.190972 30834 leader_election.cc:341] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.191962 30914 raft_consensus.cc:348] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.192655 31066 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.192688 30914 raft_consensus.cc:374] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.194099 30914 raft_consensus.cc:729] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.193212 31066 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:14:58.217638 31118 log.cc:924] Time spent T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20260806 20:14:58.218321 31066 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.182013 30756 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:14:58.220180 30835 leader_election.cc:341] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.222100 30835 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:58.223487 31009 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.224229 30990 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:58.228958 31113 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 20:14:58.216403 31120 log.cc:924] Time spent T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20260806 20:14:58.228284 30914 consensus_queue.cc:260] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.227123 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.240988 30914 ts_tablet_manager.cc:1428] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.087s user 0.020s sys 0.007s
I20260806 20:14:58.241928 30914 tablet_bootstrap.cc:492] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:58.242818 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:14:58.236846 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 75644.4us
W20260806 20:14:58.248602 30756 leader_election.cc:341] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.248687 31009 consensus_queue.cc:1035] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:58.250409 30913 tablet_bootstrap.cc:492] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:58.250878 30913 ts_tablet_manager.cc:1397] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.375s user 0.019s sys 0.008s
I20260806 20:14:58.251778 30871 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
W20260806 20:14:58.252415 31128 log.cc:924] Time spent T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260806 20:14:58.249924 31088 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.252929 30915 tablet_bootstrap.cc:654] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.236174 31113 raft_consensus.cc:686] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:58.256745 31113 consensus_queue.cc:237] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.262668 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.264298 30788 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:58.264428 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:58.281796 30913 raft_consensus.cc:348] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.282334 30913 raft_consensus.cc:374] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.282558 30913 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.282765 30602 catalog_manager.cc:5495] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:58.283097 30913 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:14:58.299474 30757 leader_election.cc:341] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.300007 30757 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:58.301481 31065 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:58.299497 30913 ts_tablet_manager.cc:1428] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.048s user 0.008s sys 0.001s
I20260806 20:14:58.305676 30603 catalog_manager.cc:5495] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:58.314822 30913 tablet_bootstrap.cc:492] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:58.305435 31088 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.331247 30913 tablet_bootstrap.cc:654] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.333381 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.336215 30914 tablet_bootstrap.cc:654] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.337430 30915 tablet_bootstrap.cc:492] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:58.337992 30915 ts_tablet_manager.cc:1397] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.320s user 0.021s sys 0.005s
I20260806 20:14:58.339072 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:58.362823 30915 raft_consensus.cc:348] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.363620 30915 raft_consensus.cc:374] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.363881 30915 raft_consensus.cc:729] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.364514 30915 consensus_queue.cc:260] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.369637 30915 ts_tablet_manager.cc:1428] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.031s user 0.004s sys 0.003s
I20260806 20:14:58.370388 30915 tablet_bootstrap.cc:492] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:58.370913 30676 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:58.374254 31088 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:58.374995 31088 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:58.375340 31088 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.382067 30995 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:58.382094 31088 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.384348 31088 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:14:58.384742 30655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:14:58.387281 30913 tablet_bootstrap.cc:492] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:58.387766 30913 ts_tablet_manager.cc:1397] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.081s user 0.015s sys 0.003s
W20260806 20:14:58.389874 30676 leader_election.cc:341] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.390179 30913 raft_consensus.cc:348] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.390853 30913 raft_consensus.cc:374] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.391135 30913 raft_consensus.cc:729] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.391714 30913 consensus_queue.cc:260] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.392402 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:58.393006 30790 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.394294 30913 ts_tablet_manager.cc:1428] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260806 20:14:58.395280 30913 tablet_bootstrap.cc:492] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:58.399523 30676 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:58.400377 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:58.401355 31122 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:14:58.410285 30915 tablet_bootstrap.cc:654] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.408646 31074 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.403486 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:58.414374 30995 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:58.420146 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:58.419688 30787 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:58.424036 30676 leader_election.cc:341] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.424054 30868 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.428524 30914 tablet_bootstrap.cc:492] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:58.428995 30914 ts_tablet_manager.cc:1397] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.187s user 0.007s sys 0.004s
I20260806 20:14:58.418541 31074 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.436858 30914 raft_consensus.cc:348] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.437698 30914 raft_consensus.cc:374] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.437960 30914 raft_consensus.cc:729] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.439018 30790 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:58.448370 31113 consensus_queue.cc:1035] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:58.438748 30914 consensus_queue.cc:260] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.441080 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.451582 30711 raft_consensus.cc:2379] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 1: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:14:58.454959 30914 ts_tablet_manager.cc:1428] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260806 20:14:58.441692 30913 tablet_bootstrap.cc:654] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.441967 31074 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:58.445762 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.456036 30914 tablet_bootstrap.cc:492] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:58.456808 30676 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:58.458194 30785 raft_consensus.cc:2379] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 1: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:14:58.458176 30915 tablet_bootstrap.cc:492] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:58.460336 30835 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:58.459250 30915 ts_tablet_manager.cc:1397] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.007s
I20260806 20:14:58.460965 31088 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:58.466317 31088 raft_consensus.cc:686] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:58.467227 31088 consensus_queue.cc:237] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.468868 31074 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.474990 30915 raft_consensus.cc:348] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.475853 30915 raft_consensus.cc:374] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.476118 30915 raft_consensus.cc:729] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.476779 30915 consensus_queue.cc:260] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.477228 31074 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:58.460708 30787 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:58.486913 30676 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:58.488314 31127 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:58.488997 31127 raft_consensus.cc:686] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:58.489830 31127 consensus_queue.cc:237] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.495659 30868 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:58.509083 31113 consensus_queue.cc:1035] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:58.511088 30915 ts_tablet_manager.cc:1428] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.047s user 0.005s sys 0.003s
I20260806 20:14:58.514675 30915 tablet_bootstrap.cc:492] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:58.515563 30914 tablet_bootstrap.cc:654] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.539502 30915 tablet_bootstrap.cc:654] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.583354 30913 tablet_bootstrap.cc:492] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:58.583832 30913 ts_tablet_manager.cc:1397] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.189s user 0.011s sys 0.007s
I20260806 20:14:58.588430 30913 raft_consensus.cc:348] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.589135 30913 raft_consensus.cc:374] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.589407 30913 raft_consensus.cc:729] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.591820 30913 consensus_queue.cc:260] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.589126 30602 catalog_manager.cc:5495] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:58.595016 30915 tablet_bootstrap.cc:492] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:58.595471 30915 ts_tablet_manager.cc:1397] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.005s
I20260806 20:14:58.595886 31112 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.596360 31112 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.596961 30913 ts_tablet_manager.cc:1428] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20260806 20:14:58.597884 30913 tablet_bootstrap.cc:492] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:58.598304 31112 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.599907 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.600639 30712 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:58.601929 30756 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:58.602846 31112 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:58.603194 31112 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:58.603758 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:58.603507 31112 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.605201 30914 tablet_bootstrap.cc:492] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:58.604406 30868 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:58.605780 30914 ts_tablet_manager.cc:1397] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.150s user 0.000s sys 0.018s
I20260806 20:14:58.606222 30913 tablet_bootstrap.cc:654] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.606993 30915 raft_consensus.cc:348] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.607684 30915 raft_consensus.cc:374] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.607942 30915 raft_consensus.cc:729] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.608534 30915 consensus_queue.cc:260] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.610460 30915 ts_tablet_manager.cc:1428] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.015s user 0.001s sys 0.005s
I20260806 20:14:58.611361 30915 tablet_bootstrap.cc:492] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:58.610229 30914 raft_consensus.cc:348] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.612969 30914 raft_consensus.cc:374] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.613300 30914 raft_consensus.cc:729] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.614065 30914 consensus_queue.cc:260] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.616678 30914 ts_tablet_manager.cc:1428] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260806 20:14:58.617564 30914 tablet_bootstrap.cc:492] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:58.617250 31112 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.621078 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:58.622442 30712 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.622014 30601 catalog_manager.cc:5495] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:58.623765 30915 tablet_bootstrap.cc:654] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.623606 30601 catalog_manager.cc:5495] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:58.628680 31112 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.629596 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:58.630159 30868 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.635921 30712 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:58.638111 30913 tablet_bootstrap.cc:492] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:58.638579 30913 ts_tablet_manager.cc:1397] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.004s
I20260806 20:14:58.638520 30868 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:58.638473 30756 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:58.640480 31112 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:58.640940 31112 raft_consensus.cc:686] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:58.641347 30913 raft_consensus.cc:348] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.642094 30913 raft_consensus.cc:374] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.642267 30914 tablet_bootstrap.cc:654] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.642364 30913 raft_consensus.cc:729] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.641922 31112 consensus_queue.cc:237] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.643170 30913 consensus_queue.cc:260] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.645529 30913 ts_tablet_manager.cc:1428] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 20:14:58.646417 30913 tablet_bootstrap.cc:492] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:58.652104 30913 tablet_bootstrap.cc:654] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.661670 30913 tablet_bootstrap.cc:492] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:58.662117 30913 ts_tablet_manager.cc:1397] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20260806 20:14:58.664472 30913 raft_consensus.cc:348] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.665354 30913 raft_consensus.cc:374] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.665656 30913 raft_consensus.cc:729] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.666307 30913 consensus_queue.cc:260] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.668725 30913 ts_tablet_manager.cc:1428] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:14:58.669677 30913 tablet_bootstrap.cc:492] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:58.670482 31074 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.670955 31074 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.673131 31074 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:58.674288 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.674587 30915 tablet_bootstrap.cc:492] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:58.674499 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.675062 30915 ts_tablet_manager.cc:1397] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.013s
I20260806 20:14:58.674966 30789 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
W20260806 20:14:58.676090 30834 leader_election.cc:341] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.676610 30835 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:58.677724 30995 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.677465 30915 raft_consensus.cc:348] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.677196 30602 catalog_manager.cc:5495] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:58.678171 30915 raft_consensus.cc:374] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.678494 30915 raft_consensus.cc:729] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.679201 30991 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:58.679142 30915 consensus_queue.cc:260] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.678182 30995 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.680897 30913 tablet_bootstrap.cc:654] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.681281 30915 ts_tablet_manager.cc:1428] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:14:58.679584 30991 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:58.681908 30991 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.682737 30995 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:58.683130 30914 tablet_bootstrap.cc:492] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:58.683506 30914 ts_tablet_manager.cc:1397] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.012s
I20260806 20:14:58.686558 30913 tablet_bootstrap.cc:492] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:58.686990 30913 ts_tablet_manager.cc:1397] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.001s
I20260806 20:14:58.685869 30914 raft_consensus.cc:348] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.687696 30914 raft_consensus.cc:374] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.687952 30914 raft_consensus.cc:729] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.688483 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.688534 30914 consensus_queue.cc:260] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.689306 30787 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:58.690881 30913 raft_consensus.cc:348] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.691556 30913 raft_consensus.cc:374] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.691797 30913 raft_consensus.cc:729] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.692449 30913 consensus_queue.cc:260] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.693928 30835 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:58.695026 30919 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:58.695652 30919 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:58.695978 30919 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.694993 30913 ts_tablet_manager.cc:1428] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:14:58.698787 30913 tablet_bootstrap.cc:492] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:58.700500 30914 ts_tablet_manager.cc:1428] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260806 20:14:58.701785 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.702243 30915 tablet_bootstrap.cc:492] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
W20260806 20:14:58.703243 30834 leader_election.cc:341] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.708828 30919 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.711498 30913 tablet_bootstrap.cc:654] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.711854 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:58.700994 30991 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:14:58.713351 30834 leader_election.cc:341] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.714119 31074 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.714586 31074 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.715723 31088 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.716225 31088 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.719442 30914 tablet_bootstrap.cc:492] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:58.720122 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:58.720815 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:58.721400 30787 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:58.722256 30834 leader_election.cc:341] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.725303 30913 tablet_bootstrap.cc:492] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:58.725776 30913 ts_tablet_manager.cc:1397] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.002s
I20260806 20:14:58.726334 31088 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.728116 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:58.728168 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.728184 30913 raft_consensus.cc:348] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.728971 30913 raft_consensus.cc:374] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.728999 30789 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:58.729342 30913 raft_consensus.cc:729] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
W20260806 20:14:58.729650 30676 leader_election.cc:341] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.729965 30913 consensus_queue.cc:260] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.730319 30676 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:58.730829 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.732664 30913 ts_tablet_manager.cc:1428] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 20:14:58.733721 30913 tablet_bootstrap.cc:492] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:58.733292 31074 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:58.734404 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.735474 30919 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:14:58.737176 30835 leader_election.cc:341] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.736449 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:58.737933 30789 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.738179 31088 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:58.738579 31088 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:58.738914 31088 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.739815 30913 tablet_bootstrap.cc:654] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.736810 30914 tablet_bootstrap.cc:654] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.740862 30991 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:58.741017 31113 consensus_queue.cc:1035] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 20:14:58.742642 30834 leader_election.cc:341] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.743050 30834 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:58.743687 30991 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:58.756197 30915 tablet_bootstrap.cc:654] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.763096 31113 consensus_queue.cc:1035] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:58.764623 31088 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.766716 30914 tablet_bootstrap.cc:492] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:58.767241 30914 ts_tablet_manager.cc:1397] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.007s
I20260806 20:14:58.769718 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:58.770090 30787 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:58.770313 31088 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.761178 30789 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:58.772545 30835 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:58.773803 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:58.770373 30790 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.774559 30991 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:58.775099 30991 raft_consensus.cc:686] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:58.777328 30991 consensus_queue.cc:237] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.778188 30914 raft_consensus.cc:348] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.778949 30914 raft_consensus.cc:374] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.779237 30914 raft_consensus.cc:729] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.779911 30914 consensus_queue.cc:260] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.783072 30869 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.782488 30914 ts_tablet_manager.cc:1428] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20260806 20:14:58.785956 30914 tablet_bootstrap.cc:492] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:58.788790 30835 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:58.789832 31074 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:58.790329 31074 raft_consensus.cc:686] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:58.791141 31074 consensus_queue.cc:237] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.796872 30869 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:58.810889 30676 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:58.812181 31088 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:58.812703 31088 raft_consensus.cc:686] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:58.813593 31113 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.813567 31088 consensus_queue.cc:237] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.814256 31113 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.816143 31113 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.817471 30790 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:58.821970 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.828153 30913 tablet_bootstrap.cc:492] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
W20260806 20:14:58.828670 30756 leader_election.cc:341] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.828751 30913 ts_tablet_manager.cc:1397] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.095s user 0.021s sys 0.013s
I20260806 20:14:58.821461 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:58.840308 30913 raft_consensus.cc:348] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.843504 30913 raft_consensus.cc:374] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.844005 30913 raft_consensus.cc:729] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.844820 30913 consensus_queue.cc:260] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.847848 30913 ts_tablet_manager.cc:1428] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260806 20:14:58.850369 30913 tablet_bootstrap.cc:492] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:58.847726 31122 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.858182 31122 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.860697 31122 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.861737 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:14:58.863190 30676 leader_election.cc:341] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.864059 30919 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:14:58.864331 30757 leader_election.cc:341] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.864769 30757 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:58.866191 31137 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:58.864113 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.864578 30919 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.869805 30919 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:14:58.872459 30676 leader_election.cc:341] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.872925 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:58.875897 31122 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:58.876113 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:58.876783 30787 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:58.878000 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.878643 30835 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:58.879002 30914 tablet_bootstrap.cc:654] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.884717 30919 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:58.885129 30919 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:58.885490 30919 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.892616 30601 catalog_manager.cc:5495] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
W20260806 20:14:58.894291 30834 leader_election.cc:341] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.894096 30601 catalog_manager.cc:5495] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:58.899821 30919 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.902472 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:14:58.903880 30834 leader_election.cc:341] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:58.908778 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:58.907732 30919 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:58.909650 30787 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.906896 30603 catalog_manager.cc:5495] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:58.910796 30915 tablet_bootstrap.cc:492] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:58.918632 30915 ts_tablet_manager.cc:1397] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.217s user 0.009s sys 0.013s
I20260806 20:14:58.921196 30787 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:58.911697 30913 tablet_bootstrap.cc:654] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.925221 30835 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:58.927258 30915 raft_consensus.cc:348] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.928067 30915 raft_consensus.cc:374] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.927269 31074 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:58.928388 30915 raft_consensus.cc:729] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.929999 31074 raft_consensus.cc:686] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:58.929776 30915 consensus_queue.cc:260] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.931289 31074 consensus_queue.cc:237] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.933648 30915 ts_tablet_manager.cc:1428] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20260806 20:14:58.934638 30915 tablet_bootstrap.cc:492] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:58.934744 30914 tablet_bootstrap.cc:492] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:58.935351 30914 ts_tablet_manager.cc:1397] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.150s user 0.012s sys 0.009s
I20260806 20:14:58.948916 30914 raft_consensus.cc:348] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.949746 30914 raft_consensus.cc:374] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.950109 30914 raft_consensus.cc:729] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.951262 30914 consensus_queue.cc:260] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.954052 30914 ts_tablet_manager.cc:1428] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260806 20:14:58.954998 30914 tablet_bootstrap.cc:492] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:58.959172 30915 tablet_bootstrap.cc:654] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.960342 30914 tablet_bootstrap.cc:654] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.966908 30913 tablet_bootstrap.cc:492] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:58.967339 30913 ts_tablet_manager.cc:1397] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.117s user 0.015s sys 0.013s
I20260806 20:14:58.969584 30913 raft_consensus.cc:348] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.970240 30868 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:58.972297 30601 catalog_manager.cc:5495] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:58.970275 30913 raft_consensus.cc:374] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.974185 30913 raft_consensus.cc:729] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.975696 30913 consensus_queue.cc:260] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.979415 30913 ts_tablet_manager.cc:1428] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260806 20:14:58.982959 30913 tablet_bootstrap.cc:492] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:58.985365 30914 tablet_bootstrap.cc:492] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:58.985869 30914 ts_tablet_manager.cc:1397] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.008s
I20260806 20:14:58.989120 30913 tablet_bootstrap.cc:654] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:58.989425 30914 raft_consensus.cc:348] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.989403 31122 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:58.990314 30914 raft_consensus.cc:374] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.990592 30914 raft_consensus.cc:729] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.992794 30915 tablet_bootstrap.cc:492] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:58.993234 30915 ts_tablet_manager.cc:1397] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.008s
I20260806 20:14:58.993763 31137 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.994285 31137 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.996073 30915 raft_consensus.cc:348] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.996454 31137 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:58.996766 30915 raft_consensus.cc:374] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:58.997076 30915 raft_consensus.cc:729] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:58.997640 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:58.997710 30915 consensus_queue.cc:260] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:58.998647 31088 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:58.999071 31088 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.000946 31088 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:14:59.001376 30756 leader_election.cc:341] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.001725 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:14:59.003345 30757 leader_election.cc:341] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.003830 30757 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:59.005229 31113 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:59.006510 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:59.007750 30915 ts_tablet_manager.cc:1428] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20260806 20:14:59.007761 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:59.008751 30915 tablet_bootstrap.cc:492] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:59.008957 30869 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:59.010993 30914 consensus_queue.cc:260] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.012331 30676 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:14:59.013249 30676 leader_election.cc:341] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.014580 31088 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:59.015170 31088 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.015767 31088 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.021322 31122 consensus_queue.cc:1035] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:59.022359 30914 ts_tablet_manager.cc:1428] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20260806 20:14:59.029844 30914 tablet_bootstrap.cc:492] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:59.032269 31113 consensus_queue.cc:1035] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:14:59.039089 31127 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:59.024647 30915 tablet_bootstrap.cc:654] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.050738 30913 tablet_bootstrap.cc:492] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:59.051235 30913 ts_tablet_manager.cc:1397] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.068s user 0.021s sys 0.033s
I20260806 20:14:59.052210 30914 tablet_bootstrap.cc:654] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.056547 31088 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.059484 30913 raft_consensus.cc:348] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.060240 30913 raft_consensus.cc:374] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.060492 30913 raft_consensus.cc:729] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.061077 30913 consensus_queue.cc:260] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.063153 30913 ts_tablet_manager.cc:1428] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
I20260806 20:14:59.064041 30913 tablet_bootstrap.cc:492] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:59.065518 31130 consensus_queue.cc:1035] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260806 20:14:59.106549 31088 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.112170 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:59.112802 30868 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.114789 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:59.114837 31113 consensus_queue.cc:1035] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 20:14:59.123432 30676 leader_election.cc:341] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.129798 30915 tablet_bootstrap.cc:492] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:59.130338 30915 ts_tablet_manager.cc:1397] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.122s user 0.040s sys 0.030s
I20260806 20:14:59.130219 30868 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:59.137318 31137 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.137918 31137 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.139345 30915 raft_consensus.cc:348] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.140064 30915 raft_consensus.cc:374] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.133404 30676 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:59.140859 30915 raft_consensus.cc:729] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.141646 30915 consensus_queue.cc:260] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.142284 30913 tablet_bootstrap.cc:654] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.144968 31127 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:59.145110 30915 ts_tablet_manager.cc:1428] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260806 20:14:59.145506 31127 raft_consensus.cc:686] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:59.146332 31127 consensus_queue.cc:237] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.151247 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:59.140192 31137 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.156284 31074 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.154891 30915 tablet_bootstrap.cc:492] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
W20260806 20:14:59.156421 30757 leader_election.cc:341] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.175822 30712 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.170953 31074 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.222667 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.233525 31074 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:59.239372 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.240640 30914 tablet_bootstrap.cc:492] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:59.241112 30914 ts_tablet_manager.cc:1397] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.212s user 0.018s sys 0.025s
I20260806 20:14:59.235574 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:59.244506 30789 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
W20260806 20:14:59.245692 30834 leader_election.cc:341] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.251960 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:59.252738 31074 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260806 20:14:59.254340 30834 consensus_peers.cc:487] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Couldn't send request to peer 42cabca48db7403b8523a3382a046043. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:59.254760 31074 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.255580 31074 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.256443 30995 consensus_queue.cc:1035] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:59.257176 31088 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.257745 31088 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.261649 31088 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.262693 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:14:59.264400 30756 leader_election.cc:341] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.265390 30995 consensus_queue.cc:1035] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:59.268633 30915 tablet_bootstrap.cc:654] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.264860 30756 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:59.269472 30914 raft_consensus.cc:348] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.270220 30914 raft_consensus.cc:374] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.270486 30914 raft_consensus.cc:729] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.270511 31137 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:59.270896 30913 tablet_bootstrap.cc:492] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:59.271464 30913 ts_tablet_manager.cc:1397] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.208s user 0.016s sys 0.019s
I20260806 20:14:59.271209 30914 consensus_queue.cc:260] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.279781 30914 ts_tablet_manager.cc:1428] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20260806 20:14:59.280720 30914 tablet_bootstrap.cc:492] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:59.283591 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:59.284787 30601 catalog_manager.cc:5495] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:59.286453 30914 tablet_bootstrap.cc:654] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.286960 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:59.286880 30913 raft_consensus.cc:348] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.288774 30913 raft_consensus.cc:374] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.289144 30913 raft_consensus.cc:729] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.299294 30913 consensus_queue.cc:260] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.300797 31088 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:14:59.300922 31137 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.301803 31137 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.303364 30913 ts_tablet_manager.cc:1428] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.032s user 0.008s sys 0.001s
I20260806 20:14:59.285269 31074 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.304327 31137 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.306260 31074 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:59.304459 30913 tablet_bootstrap.cc:492] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:59.313549 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:59.314128 30790 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.314451 30913 tablet_bootstrap.cc:654] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.315353 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:59.316402 30914 tablet_bootstrap.cc:492] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:59.316147 30871 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:59.316967 30914 ts_tablet_manager.cc:1397] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.036s user 0.021s sys 0.013s
I20260806 20:14:59.319069 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:59.320446 30914 raft_consensus.cc:348] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.321211 30914 raft_consensus.cc:374] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.321535 30914 raft_consensus.cc:729] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.322371 30914 consensus_queue.cc:260] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.324669 30914 ts_tablet_manager.cc:1428] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 20:14:59.323216 30757 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:59.326972 31113 raft_consensus.cc:2781] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:59.327303 31113 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.327600 31113 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:59.329226 30834 leader_election.cc:341] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.335198 31113 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.338374 31113 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.339828 30914 tablet_bootstrap.cc:492] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:59.342298 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:59.320683 30790 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:59.345016 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:14:59.344808 30756 leader_election.cc:341] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.350147 30919 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:59.343776 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:59.344859 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.351544 30919 raft_consensus.cc:686] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:59.353694 30871 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.355865 30919 consensus_queue.cc:237] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.367318 31074 consensus_queue.cc:1035] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:59.368933 30914 tablet_bootstrap.cc:654] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.372174 30913 tablet_bootstrap.cc:492] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:59.372630 30913 ts_tablet_manager.cc:1397] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.068s user 0.017s sys 0.018s
W20260806 20:14:59.373252 30756 leader_election.cc:341] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.375238 30913 raft_consensus.cc:348] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.375941 30913 raft_consensus.cc:374] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.376192 30913 raft_consensus.cc:729] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.376760 30913 consensus_queue.cc:260] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.380776 31127 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.381286 31127 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.384232 31127 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.384205 30915 tablet_bootstrap.cc:492] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:59.385351 30915 ts_tablet_manager.cc:1397] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.233s user 0.011s sys 0.022s
I20260806 20:14:59.389127 30915 raft_consensus.cc:348] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.389917 30915 raft_consensus.cc:374] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.389825 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:59.390278 30915 raft_consensus.cc:729] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.390827 30789 raft_consensus.cc:2379] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 1: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:14:59.390590 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:59.390956 30915 consensus_queue.cc:260] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.392505 30676 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:14:59.393497 31088 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.384202 30913 ts_tablet_manager.cc:1428] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260806 20:14:59.396201 30913 tablet_bootstrap.cc:492] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:59.406267 30915 ts_tablet_manager.cc:1428] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260806 20:14:59.407418 30915 tablet_bootstrap.cc:492] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:59.413172 30913 tablet_bootstrap.cc:654] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.423094 31088 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:59.389833 31137 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.423960 31137 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.426152 31137 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.429255 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:14:59.430682 30757 leader_election.cc:341] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.432233 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.432934 30712 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:14:59.435380 30756 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:59.436867 31113 raft_consensus.cc:2781] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:59.437165 31113 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.437501 31113 raft_consensus.cc:3037] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.416121 30871 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:59.443423 30757 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:59.440516 30603 catalog_manager.cc:5495] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:59.438882 31130 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260806 20:14:59.450541 31127 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:59.451011 30914 tablet_bootstrap.cc:492] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:59.451540 30914 ts_tablet_manager.cc:1397] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.112s user 0.036s sys 0.024s
I20260806 20:14:59.447911 31137 raft_consensus.cc:2781] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:59.456931 31137 raft_consensus.cc:686] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:59.466161 30914 raft_consensus.cc:348] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.467307 30914 raft_consensus.cc:374] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.467653 30914 raft_consensus.cc:729] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.471498 31137 consensus_queue.cc:237] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.476886 30914 consensus_queue.cc:260] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.468603 31113 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.482398 30915 tablet_bootstrap.cc:654] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.485558 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:59.487623 31113 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:14:59.487131 30757 leader_election.cc:341] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:14:59.492636 30655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:14:59.486918 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:59.494256 30713 raft_consensus.cc:3037] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.496707 30913 tablet_bootstrap.cc:492] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:59.497164 30913 ts_tablet_manager.cc:1397] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.101s user 0.027s sys 0.011s
I20260806 20:14:59.498977 30919 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:59.499857 30913 raft_consensus.cc:348] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.500552 30913 raft_consensus.cc:374] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.500824 30913 raft_consensus.cc:729] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.503204 30914 ts_tablet_manager.cc:1428] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.051s user 0.010s sys 0.000s
I20260806 20:14:59.504108 30914 tablet_bootstrap.cc:492] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:59.501434 30913 consensus_queue.cc:260] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.515131 30919 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.515650 30919 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.518644 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:59.519297 30787 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:59.520473 30835 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:59.522413 30919 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:59.523442 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.523622 30713 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:59.524125 30711 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:59.527873 30756 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:59.529968 31113 raft_consensus.cc:2781] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:59.527985 30990 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:59.530517 31113 raft_consensus.cc:686] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:59.531908 30990 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.532024 30913 ts_tablet_manager.cc:1428] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.035s user 0.005s sys 0.002s
I20260806 20:14:59.532294 30990 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.534098 30913 tablet_bootstrap.cc:492] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:59.532680 31113 consensus_queue.cc:237] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.569275 30914 tablet_bootstrap.cc:654] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.572714 30991 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:59.576116 30603 catalog_manager.cc:5495] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:59.582142 31088 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:59.586258 30915 tablet_bootstrap.cc:492] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:59.586867 30915 ts_tablet_manager.cc:1397] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.180s user 0.006s sys 0.012s
I20260806 20:14:59.591603 30915 raft_consensus.cc:348] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.592471 30915 raft_consensus.cc:374] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.592859 30915 raft_consensus.cc:729] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.593590 30915 consensus_queue.cc:260] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.598412 30915 ts_tablet_manager.cc:1428] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260806 20:14:59.603339 31112 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.604153 31112 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.593128 30990 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.606745 30990 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:59.607920 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:59.608597 30712 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.611487 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:59.610356 30915 tablet_bootstrap.cc:492] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:59.612812 30789 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.614180 30913 tablet_bootstrap.cc:654] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.611878 31112 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.617025 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:14:59.619280 30757 leader_election.cc:341] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.620324 30914 tablet_bootstrap.cc:492] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:59.620795 30914 ts_tablet_manager.cc:1397] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.117s user 0.019s sys 0.001s
I20260806 20:14:59.621497 30712 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:59.622102 31122 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.623494 31122 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.623970 30834 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:14:59.624953 30990 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:59.625582 31122 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.625736 30990 raft_consensus.cc:686] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:59.627772 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.629957 30990 consensus_queue.cc:237] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.631809 30995 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.632434 30995 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.634380 30995 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:59.627748 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:14:59.635823 30756 leader_election.cc:341] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.636361 30756 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:59.637632 31065 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:14:59.638449 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:59.638640 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:59.639374 30785 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:59.641816 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:14:59.643554 30834 leader_election.cc:341] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.645159 30835 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:59.645530 30790 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
W20260806 20:14:59.645707 30676 leader_election.cc:341] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.646437 30995 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260806 20:14:59.646598 30676 consensus_peers.cc:487] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 -> Peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Couldn't send request to peer d4181d4d907f477f9feddc44ba3b9207. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:59.646828 30995 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.647249 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:59.647429 31112 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.647881 31112 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.648236 31088 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:59.648684 31088 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.649263 31088 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.649830 31112 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.647184 30995 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.650594 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.651238 30712 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:59.652343 30756 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:59.653673 30789 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:59.655700 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:59.656359 30871 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:14:59.653241 31112 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:14:59.657582 31112 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.657922 31112 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:59.664733 30915 tablet_bootstrap.cc:654] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.667343 30914 raft_consensus.cc:348] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.668135 30914 raft_consensus.cc:374] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.668397 30914 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.669036 30914 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.674654 31088 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.681985 31112 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.684756 31112 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.684710 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:59.685431 30712 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:59.685429 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:59.686147 30787 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.682055 30914 ts_tablet_manager.cc:1428] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.061s user 0.008s sys 0.000s
I20260806 20:14:59.689815 31088 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.695711 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:14:59.696372 30868 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:14:59.677911 30995 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.698333 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:14:59.699652 30676 leader_election.cc:341] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.698945 30712 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:14:59.700973 30914 tablet_bootstrap.cc:492] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:59.701220 30756 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:59.702316 31113 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:14:59.702839 31113 raft_consensus.cc:686] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:14:59.703626 31113 consensus_queue.cc:237] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.706645 30913 tablet_bootstrap.cc:492] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:59.707094 30913 ts_tablet_manager.cc:1397] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.173s user 0.012s sys 0.024s
I20260806 20:14:59.707901 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:59.708520 30789 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.710520 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:59.711319 30913 raft_consensus.cc:348] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.711992 30913 raft_consensus.cc:374] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 20:14:59.712092 30834 leader_election.cc:341] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.712283 30913 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.712980 30995 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:59.713164 30913 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.717500 30913 ts_tablet_manager.cc:1428] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20260806 20:14:59.718405 30913 tablet_bootstrap.cc:492] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:59.719177 30789 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:59.710832 30787 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:14:59.721606 30835 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:59.692035 30600 catalog_manager.cc:5495] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:59.722301 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:14:59.722841 30995 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:59.723321 30995 raft_consensus.cc:686] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:14:59.723960 31122 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:59.724110 30995 consensus_queue.cc:237] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.724807 31122 raft_consensus.cc:686] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:14:59.725732 31122 consensus_queue.cc:237] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.728243 30868 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:14:59.730816 30915 tablet_bootstrap.cc:492] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:59.734917 30915 ts_tablet_manager.cc:1397] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.125s user 0.016s sys 0.000s
I20260806 20:14:59.738466 30913 tablet_bootstrap.cc:654] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.743711 30603 catalog_manager.cc:5495] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:59.746634 30914 tablet_bootstrap.cc:654] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.751137 30915 raft_consensus.cc:348] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.751781 30915 raft_consensus.cc:374] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.751982 30915 raft_consensus.cc:729] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.752449 30915 consensus_queue.cc:260] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.759290 30915 ts_tablet_manager.cc:1428] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.024s user 0.004s sys 0.001s
I20260806 20:14:59.760228 30915 tablet_bootstrap.cc:492] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:59.766134 30915 tablet_bootstrap.cc:654] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.756806 30602 catalog_manager.cc:5495] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:59.775767 30914 tablet_bootstrap.cc:492] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:59.776244 30914 ts_tablet_manager.cc:1397] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.009s
I20260806 20:14:59.777936 30601 catalog_manager.cc:5495] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:59.778662 30914 raft_consensus.cc:348] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.779342 30914 raft_consensus.cc:374] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.779596 30914 raft_consensus.cc:729] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.780225 30914 consensus_queue.cc:260] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.782989 30914 ts_tablet_manager.cc:1428] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 20:14:59.783870 30914 tablet_bootstrap.cc:492] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
W20260806 20:14:59.791860 30834 consensus_peers.cc:487] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Couldn't send request to peer 42cabca48db7403b8523a3382a046043. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:59.795363 30915 tablet_bootstrap.cc:492] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:59.795995 30915 ts_tablet_manager.cc:1397] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.002s
I20260806 20:14:59.798404 30915 raft_consensus.cc:348] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.799155 30915 raft_consensus.cc:374] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.799459 30915 raft_consensus.cc:729] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.800110 30915 consensus_queue.cc:260] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.800982 30914 tablet_bootstrap.cc:654] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.802613 30915 ts_tablet_manager.cc:1428] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:14:59.803773 30915 tablet_bootstrap.cc:492] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:59.803647 30600 catalog_manager.cc:5495] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 2, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:14:59.809629 30915 tablet_bootstrap.cc:654] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.810940 30913 tablet_bootstrap.cc:492] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:59.811498 30913 ts_tablet_manager.cc:1397] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.093s user 0.017s sys 0.032s
I20260806 20:14:59.815335 30913 raft_consensus.cc:348] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.816128 30913 raft_consensus.cc:374] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.816430 30913 raft_consensus.cc:729] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.817153 30913 consensus_queue.cc:260] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.816128 30990 consensus_queue.cc:1035] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 20:14:59.819600 30913 ts_tablet_manager.cc:1428] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260806 20:14:59.820545 30913 tablet_bootstrap.cc:492] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:59.829176 30914 tablet_bootstrap.cc:492] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:14:59.829802 30914 ts_tablet_manager.cc:1397] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.014s
I20260806 20:14:59.832644 30914 raft_consensus.cc:348] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.835255 30913 tablet_bootstrap.cc:654] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.833400 30914 raft_consensus.cc:374] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.835927 30914 raft_consensus.cc:729] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.836563 30914 consensus_queue.cc:260] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.838953 30914 ts_tablet_manager.cc:1428] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 20:14:59.839942 30914 tablet_bootstrap.cc:492] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:14:59.849418 30915 tablet_bootstrap.cc:492] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:59.849956 30915 ts_tablet_manager.cc:1397] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.046s user 0.016s sys 0.000s
I20260806 20:14:59.857697 30990 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.858243 30990 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.860438 30990 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:59.860805 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.860497 30915 raft_consensus.cc:348] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.861603 30915 raft_consensus.cc:374] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.861891 30915 raft_consensus.cc:729] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
W20260806 20:14:59.862272 30834 leader_election.cc:341] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.862639 30915 consensus_queue.cc:260] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.865183 30915 ts_tablet_manager.cc:1428] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260806 20:14:59.865908 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:59.866649 30787 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:59.867847 30915 tablet_bootstrap.cc:492] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:59.869074 30835 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:14:59.870707 30995 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:59.871050 30995 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.871428 30995 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.877605 30914 tablet_bootstrap.cc:654] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.882870 30913 tablet_bootstrap.cc:492] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:59.883426 30913 ts_tablet_manager.cc:1397] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.013s
I20260806 20:14:59.886965 30913 raft_consensus.cc:348] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.887647 30913 raft_consensus.cc:374] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.887914 30913 raft_consensus.cc:729] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.888491 30913 consensus_queue.cc:260] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.890316 30915 tablet_bootstrap.cc:654] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.891165 30913 ts_tablet_manager.cc:1428] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260806 20:14:59.892544 30913 tablet_bootstrap.cc:492] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:59.898516 30913 tablet_bootstrap.cc:654] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.904737 30915 tablet_bootstrap.cc:492] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:14:59.905262 30915 ts_tablet_manager.cc:1397] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.002s
I20260806 20:14:59.887434 30995 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.913938 30995 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:59.914990 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:14:59.915601 30789 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.918032 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:59.921911 30789 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:14:59.922987 30835 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:59.923483 30913 tablet_bootstrap.cc:492] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:14:59.923480 30915 raft_consensus.cc:348] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.924396 31074 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:14:59.924904 31074 raft_consensus.cc:686] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:14:59.926239 30834 leader_election.cc:341] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.925824 31074 consensus_queue.cc:237] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.924475 30915 raft_consensus.cc:374] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.928467 30915 raft_consensus.cc:729] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.924050 30913 ts_tablet_manager.cc:1397] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.000s
I20260806 20:14:59.929167 30915 consensus_queue.cc:260] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.931565 30913 raft_consensus.cc:348] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.934966 30913 raft_consensus.cc:374] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:14:59.935443 30913 raft_consensus.cc:729] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:14:59.937255 30913 consensus_queue.cc:260] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.941440 30915 ts_tablet_manager.cc:1428] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20260806 20:14:59.942461 30915 tablet_bootstrap.cc:492] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:14:59.947696 30915 tablet_bootstrap.cc:654] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.948827 30913 ts_tablet_manager.cc:1428] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260806 20:14:59.949823 30913 tablet_bootstrap.cc:492] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:14:59.951651 31088 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.952162 31088 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.956398 30913 tablet_bootstrap.cc:654] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:14:59.961772 30603 catalog_manager.cc:5495] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:14:59.964910 30991 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.965415 30991 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.967406 30991 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:14:59.970731 30712 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:14:59.974431 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:59.975083 30789 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:59.976238 30835 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:59.978109 31088 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:14:59.978293 31074 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:59.978036 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.978648 31074 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.978994 31074 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:59.979493 30757 consensus_peers.cc:487] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 -> Peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Couldn't send request to peer 58d59beb597f4ed99d84d102ddb41ff5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:14:59.982259 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:14:59.982868 30789 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:59.984907 31112 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:14:59.983214 31113 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:14:59.985493 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:14:59.986217 30871 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:14:59.986060 31113 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.985505 30676 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:14:59.978963 30713 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:14:59.994091 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:14:59.992959 31127 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:14:59.995702 31127 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:14:59.995507 31074 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:14:59.996045 31127 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:14:59.996686 30756 leader_election.cc:341] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:14:59.998198 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:14:59.998791 30711 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.001638 31113 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:00.015990 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:00.018532 30757 leader_election.cc:341] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:00.019017 30757 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:00.022408 31065 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:00.023175 31065 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:00.029117 30711 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:00.030110 31074 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:00.030401 30834 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:00.031232 30995 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:00.033967 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:00.034439 30789 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.038552 30915 tablet_bootstrap.cc:492] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:00.039261 30915 ts_tablet_manager.cc:1397] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.097s user 0.014s sys 0.007s
I20260806 20:15:00.031803 30995 raft_consensus.cc:686] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:00.043006 30995 consensus_queue.cc:237] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.038497 31113 consensus_queue.cc:1035] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260806 20:15:00.047266 30915 raft_consensus.cc:348] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.047955 30915 raft_consensus.cc:374] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.048211 30915 raft_consensus.cc:729] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.057647 31088 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:00.048902 30915 consensus_queue.cc:260] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.061565 30914 tablet_bootstrap.cc:492] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:00.062106 30914 ts_tablet_manager.cc:1397] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.222s user 0.028s sys 0.032s
I20260806 20:15:00.065893 30913 tablet_bootstrap.cc:492] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:00.066401 30913 ts_tablet_manager.cc:1397] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.117s user 0.021s sys 0.048s
I20260806 20:15:00.058372 31088 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.086726 30789 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:00.074337 31127 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.097839 30914 raft_consensus.cc:348] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.098563 30914 raft_consensus.cc:374] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.098830 30914 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.099431 30914 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.104491 31088 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:00.106988 30913 raft_consensus.cc:348] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.107834 30913 raft_consensus.cc:374] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.108101 30913 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.110375 30914 ts_tablet_manager.cc:1428] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.048s user 0.003s sys 0.004s
I20260806 20:15:00.107075 30601 catalog_manager.cc:5495] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:00.114022 30913 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.117969 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:00.121804 31127 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:00.128101 30914 tablet_bootstrap.cc:492] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:00.129699 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:00.130370 30871 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.131500 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:00.132148 30787 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.133695 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:00.137318 30913 ts_tablet_manager.cc:1428] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.071s user 0.007s sys 0.000s
I20260806 20:15:00.138262 30913 tablet_bootstrap.cc:492] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:00.142208 30915 ts_tablet_manager.cc:1428] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.103s user 0.014s sys 0.010s
I20260806 20:15:00.142300 30676 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:00.144963 31122 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:00.148384 30915 tablet_bootstrap.cc:492] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:00.151297 30995 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:00.153857 30919 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:00.154328 30919 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.156319 30919 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:00.158058 30914 tablet_bootstrap.cc:654] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.158246 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:00.159052 30871 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
W20260806 20:15:00.160221 30834 leader_election.cc:341] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:00.160679 30676 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:00.161466 31127 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:00.161900 31127 raft_consensus.cc:686] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:00.162057 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:00.162624 30789 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:15:00.163519 30835 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:00.165838 30995 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:00.166230 30995 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:00.165839 30919 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:00.166894 30995 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.169515 30915 tablet_bootstrap.cc:654] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.167640 31127 consensus_queue.cc:237] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.174546 30871 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 20:15:00.187266 31233 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:00.191660 31074 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:00.192211 31074 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.195750 30709 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.195744 31074 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:00.199855 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:00.203523 30834 leader_election.cc:341] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:00.205925 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:00.207968 30835 leader_election.cc:341] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:00.208388 30835 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:00.209808 30991 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:00.205307 31234 consensus_queue.cc:1035] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260806 20:15:00.222162 30995 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.224170 30995 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:00.228688 31233 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:00.232342 30676 consensus_peers.cc:487] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 -> Peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Couldn't send request to peer 58d59beb597f4ed99d84d102ddb41ff5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:00.214118 30787 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:00.233314 31233 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.262863 30913 tablet_bootstrap.cc:654] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.245615 30602 catalog_manager.cc:5495] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:00.350450 31233 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:00.353026 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 119031us
I20260806 20:15:00.358408 30713 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.363437 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:00.298210 31238 log.cc:924] Time spent T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260806 20:15:00.241430 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:00.369510 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:00.371194 30914 tablet_bootstrap.cc:492] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:00.371774 30914 ts_tablet_manager.cc:1397] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.255s user 0.022s sys 0.022s
I20260806 20:15:00.371685 31088 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:00.372260 31088 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.371189 30707 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 20:15:00.379025 31088 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:00.379200 30914 raft_consensus.cc:348] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.379935 30914 raft_consensus.cc:374] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.380227 30914 raft_consensus.cc:729] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.380874 30914 consensus_queue.cc:260] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.383414 30914 ts_tablet_manager.cc:1428] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 20:15:00.384775 30914 tablet_bootstrap.cc:492] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:00.389314 30913 tablet_bootstrap.cc:492] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:00.389889 30913 ts_tablet_manager.cc:1397] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.252s user 0.008s sys 0.008s
I20260806 20:15:00.233230 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:00.390945 30790 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:15:00.394009 30834 consensus_peers.cc:487] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Couldn't send request to peer 42cabca48db7403b8523a3382a046043. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:00.397744 31233 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:00.398761 30913 raft_consensus.cc:348] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:00.404295 30834 leader_election.cc:341] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:00.431917 30913 raft_consensus.cc:374] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.432274 30913 raft_consensus.cc:729] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.432883 30913 consensus_queue.cc:260] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.434338 31234 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:00.435024 30913 ts_tablet_manager.cc:1428] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.045s user 0.007s sys 0.001s
I20260806 20:15:00.434912 31234 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.437526 30913 tablet_bootstrap.cc:492] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
W20260806 20:15:00.437300 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 64761.8us
I20260806 20:15:00.440443 30757 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:00.448451 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:00.441879 31113 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:00.451813 30789 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.453001 31234 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:00.465211 30914 tablet_bootstrap.cc:654] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.465242 30915 tablet_bootstrap.cc:492] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:00.467339 30915 ts_tablet_manager.cc:1397] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.324s user 0.011s sys 0.014s
I20260806 20:15:00.455966 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:00.479172 31074 consensus_queue.cc:1035] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:00.477321 30787 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:15:00.483721 30676 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:00.484740 31234 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:00.485101 31234 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:00.485410 31234 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.490734 30790 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:00.491770 30914 tablet_bootstrap.cc:492] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:00.492233 30914 ts_tablet_manager.cc:1397] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.108s user 0.007s sys 0.007s
I20260806 20:15:00.492590 30835 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:00.494686 30914 raft_consensus.cc:348] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.495405 30914 raft_consensus.cc:374] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.495738 30914 raft_consensus.cc:729] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.496457 30914 consensus_queue.cc:260] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.498921 30914 ts_tablet_manager.cc:1428] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260806 20:15:00.500129 30914 tablet_bootstrap.cc:492] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:00.495061 31074 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:00.501895 31074 raft_consensus.cc:686] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:00.502918 31074 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.505910 30914 tablet_bootstrap.cc:654] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.508474 31234 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.483387 30915 raft_consensus.cc:348] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.513389 30915 raft_consensus.cc:374] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.513712 30915 raft_consensus.cc:729] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.514318 30915 consensus_queue.cc:260] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.518800 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:00.519426 30788 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.525983 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:00.531973 30913 tablet_bootstrap.cc:654] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.532809 30915 ts_tablet_manager.cc:1428] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.065s user 0.004s sys 0.002s
I20260806 20:15:00.533903 30915 tablet_bootstrap.cc:492] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
W20260806 20:15:00.554574 30834 consensus_peers.cc:487] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Couldn't send request to peer 42cabca48db7403b8523a3382a046043. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:00.535601 31234 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:00.556782 30676 leader_election.cc:341] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:00.572322 30602 catalog_manager.cc:5495] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:00.567386 30991 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260806 20:15:00.594571 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:00.602205 31245 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:00.602866 31245 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.606899 30995 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:00.607599 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:00.607194 31245 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:00.626921 31241 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 20:15:00.665596 30915 tablet_bootstrap.cc:654] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
W20260806 20:15:00.672135 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 104071us
I20260806 20:15:00.671639 30788 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:00.678735 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:00.680889 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:00.714823 30914 tablet_bootstrap.cc:492] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:00.715132 31253 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:00.718906 30914 ts_tablet_manager.cc:1397] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.219s user 0.053s sys 0.083s
I20260806 20:15:00.721661 30914 raft_consensus.cc:348] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.722455 30914 raft_consensus.cc:374] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.722723 30914 raft_consensus.cc:729] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.723349 30914 consensus_queue.cc:260] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.729424 30914 ts_tablet_manager.cc:1428] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260806 20:15:00.724040 30676 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:00.738644 31234 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:00.739132 31234 raft_consensus.cc:686] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:00.739945 31234 consensus_queue.cc:237] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.746075 30914 tablet_bootstrap.cc:492] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:00.748544 30707 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:15:00.752405 30676 leader_election.cc:341] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:00.754388 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:00.756052 31088 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:00.756858 30913 tablet_bootstrap.cc:492] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:00.757331 30913 ts_tablet_manager.cc:1397] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.320s user 0.019s sys 0.004s
I20260806 20:15:00.760028 30914 tablet_bootstrap.cc:654] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.757607 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:00.762442 30915 tablet_bootstrap.cc:492] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:00.764871 30915 ts_tablet_manager.cc:1397] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.231s user 0.011s sys 0.002s
I20260806 20:15:00.772957 30869 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.778498 30915 raft_consensus.cc:348] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.780400 30915 raft_consensus.cc:374] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.780679 30915 raft_consensus.cc:729] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.781286 30915 consensus_queue.cc:260] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.796129 30913 raft_consensus.cc:348] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.800621 30913 raft_consensus.cc:374] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.800913 30913 raft_consensus.cc:729] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.801530 30913 consensus_queue.cc:260] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.810004 30915 ts_tablet_manager.cc:1428] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.045s user 0.006s sys 0.000s
I20260806 20:15:00.811000 30915 tablet_bootstrap.cc:492] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:00.812517 31253 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:00.819885 30913 ts_tablet_manager.cc:1428] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20260806 20:15:00.821019 30913 tablet_bootstrap.cc:492] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:00.829394 30603 catalog_manager.cc:5495] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:00.846889 30915 tablet_bootstrap.cc:654] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.849349 30914 tablet_bootstrap.cc:492] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:00.852685 30914 ts_tablet_manager.cc:1397] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.107s user 0.021s sys 0.034s
I20260806 20:15:00.855860 30914 raft_consensus.cc:348] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.856700 30914 raft_consensus.cc:374] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.857023 30914 raft_consensus.cc:729] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.857766 30914 consensus_queue.cc:260] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.859951 30914 ts_tablet_manager.cc:1428] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:15:00.860910 30914 tablet_bootstrap.cc:492] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:00.866237 30871 raft_consensus.cc:3037] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.876729 30913 tablet_bootstrap.cc:654] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.881781 31088 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 20:15:00.887291 30655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:15:00.908172 30914 tablet_bootstrap.cc:654] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.911281 30915 tablet_bootstrap.cc:492] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:00.911870 30915 ts_tablet_manager.cc:1397] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.101s user 0.020s sys 0.010s
I20260806 20:15:00.913497 30913 tablet_bootstrap.cc:492] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:00.914016 30913 ts_tablet_manager.cc:1397] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.012s
I20260806 20:15:00.915319 30915 raft_consensus.cc:348] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.916098 30915 raft_consensus.cc:374] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.916379 30915 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.916640 30913 raft_consensus.cc:348] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.917326 30913 raft_consensus.cc:374] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.918293 30913 raft_consensus.cc:729] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.918905 30995 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:00.918978 30913 consensus_queue.cc:260] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.917018 30915 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.923266 30913 ts_tablet_manager.cc:1428] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 20:15:00.923524 30915 ts_tablet_manager.cc:1428] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260806 20:15:00.924496 30915 tablet_bootstrap.cc:492] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:00.924777 30913 tablet_bootstrap.cc:492] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:00.925715 31232 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:00.926223 31232 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.928201 31232 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:00.929800 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:00.930241 30915 tablet_bootstrap.cc:654] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.930544 30707 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:15:00.932554 30913 tablet_bootstrap.cc:654] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.932981 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:00.933627 30871 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:15:00.936693 30756 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:00.938545 31232 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:00.938973 31232 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:00.939486 31232 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.942757 30915 tablet_bootstrap.cc:492] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:00.943252 30915 ts_tablet_manager.cc:1397] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20260806 20:15:00.945801 30915 raft_consensus.cc:348] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.946521 30915 raft_consensus.cc:374] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.946771 30915 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.952147 30915 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.954603 30915 ts_tablet_manager.cc:1428] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260806 20:15:00.955509 30915 tablet_bootstrap.cc:492] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:00.956923 31232 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.960166 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:00.960757 30707 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.961395 30915 tablet_bootstrap.cc:654] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.963848 30913 tablet_bootstrap.cc:492] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:00.964309 30913 ts_tablet_manager.cc:1397] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.010s
I20260806 20:15:00.968819 30713 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.969292 30913 raft_consensus.cc:348] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.970064 30913 raft_consensus.cc:374] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.970322 30913 raft_consensus.cc:729] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.971094 30707 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:00.971170 30913 consensus_queue.cc:260] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.972954 30756 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:00.973502 30913 ts_tablet_manager.cc:1428] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
I20260806 20:15:00.973985 31233 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:00.974326 30915 tablet_bootstrap.cc:492] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:00.974426 30913 tablet_bootstrap.cc:492] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:00.974803 30915 ts_tablet_manager.cc:1397] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260806 20:15:00.974478 31233 raft_consensus.cc:686] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:00.975775 31233 consensus_queue.cc:237] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.978086 30995 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:00.978358 30915 raft_consensus.cc:348] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.979862 30915 raft_consensus.cc:374] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:00.980124 30915 raft_consensus.cc:729] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:00.980701 30915 consensus_queue.cc:260] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.985515 30913 tablet_bootstrap.cc:654] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:00.987566 31232 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:00.988725 30915 ts_tablet_manager.cc:1428] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20260806 20:15:00.988880 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:00.990131 30915 tablet_bootstrap.cc:492] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:00.990248 30865 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:00.993957 30995 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:00.994457 30995 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.995292 31232 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:00.995764 31232 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:00.997742 31232 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:00.998483 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:00.999208 30788 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:15:01.000777 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.001327 30835 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:01.002540 30915 tablet_bootstrap.cc:654] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
W20260806 20:15:01.003077 30756 leader_election.cc:341] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.005618 30995 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:01.023392 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:01.027024 30995 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.029381 30995 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.025200 30991 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:01.038374 30995 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:01.028872 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:01.043061 30834 leader_election.cc:341] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.033962 30991 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:01.045048 30991 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.047250 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.050434 30712 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:15:01.059147 30834 leader_election.cc:341] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.064610 30991 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.066133 30865 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:01.087059 30602 catalog_manager.cc:5495] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:01.089576 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:01.104943 30835 leader_election.cc:341] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.105342 30835 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:01.106323 30995 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:01.107182 30871 raft_consensus.cc:2379] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 1: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:01.115011 30914 tablet_bootstrap.cc:492] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:01.115525 30914 ts_tablet_manager.cc:1397] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.255s user 0.026s sys 0.047s
I20260806 20:15:01.116495 30757 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:01.117738 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:01.117942 31065 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:15:01.120781 30834 leader_election.cc:341] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.123006 30913 tablet_bootstrap.cc:492] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:01.124413 30913 ts_tablet_manager.cc:1397] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.150s user 0.028s sys 0.059s
I20260806 20:15:01.123389 30991 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:01.125912 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:01.125962 30914 raft_consensus.cc:348] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.128650 30914 raft_consensus.cc:374] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.128935 30914 raft_consensus.cc:729] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.127672 30915 tablet_bootstrap.cc:492] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:01.129731 30915 ts_tablet_manager.cc:1397] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.140s user 0.010s sys 0.010s
I20260806 20:15:01.131561 30914 consensus_queue.cc:260] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.133786 30914 ts_tablet_manager.cc:1428] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260806 20:15:01.135452 30914 tablet_bootstrap.cc:492] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:01.131978 30913 raft_consensus.cc:348] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.136255 30913 raft_consensus.cc:374] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.136579 30913 raft_consensus.cc:729] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.139053 31065 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:01.140597 30914 tablet_bootstrap.cc:654] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.142452 30915 raft_consensus.cc:348] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.143349 30915 raft_consensus.cc:374] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.143658 30915 raft_consensus.cc:729] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.144347 30915 consensus_queue.cc:260] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.139644 30913 consensus_queue.cc:260] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.147331 30915 ts_tablet_manager.cc:1428] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260806 20:15:01.148941 30913 ts_tablet_manager.cc:1428] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.021s user 0.003s sys 0.002s
I20260806 20:15:01.149938 30915 tablet_bootstrap.cc:492] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:01.150878 30788 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:01.150944 31130 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:01.157917 30913 tablet_bootstrap.cc:492] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:01.164697 30835 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:01.166836 31254 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:01.167317 31254 raft_consensus.cc:686] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:01.177101 30913 tablet_bootstrap.cc:654] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.177347 30915 tablet_bootstrap.cc:654] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.177497 31254 consensus_queue.cc:237] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.200713 31065 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.201162 30914 tablet_bootstrap.cc:492] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:01.201675 30914 ts_tablet_manager.cc:1397] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.015s
I20260806 20:15:01.201195 31065 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.214752 31065 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.215706 30913 tablet_bootstrap.cc:492] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:01.216267 30913 ts_tablet_manager.cc:1397] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.010s
I20260806 20:15:01.219866 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:01.223235 30757 leader_election.cc:341] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.225071 30913 raft_consensus.cc:348] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.225855 30913 raft_consensus.cc:374] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.226131 30913 raft_consensus.cc:729] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.228349 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.229015 30713 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:15:01.230203 30756 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:01.231822 31065 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:01.232187 31065 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:01.232503 31065 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.232834 30913 consensus_queue.cc:260] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.238741 30913 ts_tablet_manager.cc:1428] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260806 20:15:01.242305 30913 tablet_bootstrap.cc:492] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:01.241962 31065 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.245167 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:01.245805 30713 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.254405 30914 raft_consensus.cc:348] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.255255 30914 raft_consensus.cc:374] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.255573 30914 raft_consensus.cc:729] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.256362 30914 consensus_queue.cc:260] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.251415 30991 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:01.260478 30676 consensus_peers.cc:487] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 -> Peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Couldn't send request to peer 58d59beb597f4ed99d84d102ddb41ff5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:01.258719 30713 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:01.262475 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:01.259987 30991 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.267895 30914 ts_tablet_manager.cc:1428] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.066s user 0.004s sys 0.009s
I20260806 20:15:01.268567 30914 tablet_bootstrap.cc:492] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:01.269986 30915 tablet_bootstrap.cc:492] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:01.270447 30915 ts_tablet_manager.cc:1397] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.121s user 0.024s sys 0.013s
I20260806 20:15:01.271956 30991 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:01.274243 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:01.265668 31065 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.271875 30756 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:01.278501 30757 leader_election.cc:341] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.280714 31233 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:01.281255 31233 raft_consensus.cc:686] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:01.282121 31233 consensus_queue.cc:237] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.276604 30602 catalog_manager.cc:5495] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:01.284435 30915 raft_consensus.cc:348] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:01.276561 30835 leader_election.cc:341] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.288362 30913 tablet_bootstrap.cc:654] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.301354 31130 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.317963 31130 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.318822 31254 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.301074 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.319327 31254 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.320031 31130 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.321383 31254 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:01.321504 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:01.288666 30915 raft_consensus.cc:374] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.322357 30915 raft_consensus.cc:729] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
W20260806 20:15:01.322623 30834 leader_election.cc:341] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.323076 30834 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:01.323410 30676 leader_election.cc:341] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.323010 30915 consensus_queue.cc:260] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.292490 30914 tablet_bootstrap.cc:654] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.324883 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.325655 30915 ts_tablet_manager.cc:1428] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.055s user 0.004s sys 0.004s
I20260806 20:15:01.326771 30915 tablet_bootstrap.cc:492] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:01.328363 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:01.329749 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:01.331349 30676 leader_election.cc:341] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.331777 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:01.331338 31254 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:01.334395 30915 tablet_bootstrap.cc:654] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.335008 30913 tablet_bootstrap.cc:492] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:01.338050 30913 ts_tablet_manager.cc:1397] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.005s
I20260806 20:15:01.339753 31245 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:01.340992 30602 catalog_manager.cc:5495] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:01.342486 30913 raft_consensus.cc:348] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.343703 30913 raft_consensus.cc:374] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.344013 30913 raft_consensus.cc:729] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.344761 30913 consensus_queue.cc:260] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.346755 30913 ts_tablet_manager.cc:1428] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:15:01.348515 30914 tablet_bootstrap.cc:492] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:01.348649 30913 tablet_bootstrap.cc:492] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:01.332353 30835 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:01.353199 30915 tablet_bootstrap.cc:492] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:01.353756 30915 ts_tablet_manager.cc:1397] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.001s
I20260806 20:15:01.355787 30914 ts_tablet_manager.cc:1397] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.087s user 0.013s sys 0.005s
I20260806 20:15:01.355808 31254 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:01.358460 30914 raft_consensus.cc:348] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.359174 30914 raft_consensus.cc:374] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.359450 30914 raft_consensus.cc:729] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.360064 30914 consensus_queue.cc:260] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.360607 30915 raft_consensus.cc:348] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.361320 30915 raft_consensus.cc:374] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.362365 30914 ts_tablet_manager.cc:1428] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 20:15:01.363307 30914 tablet_bootstrap.cc:492] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:01.366179 30913 tablet_bootstrap.cc:654] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.361588 30915 raft_consensus.cc:729] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.369352 30915 consensus_queue.cc:260] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.369923 30914 tablet_bootstrap.cc:654] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.371672 30915 ts_tablet_manager.cc:1428] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260806 20:15:01.372108 31065 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.372591 31065 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.374603 31065 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.376535 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:01.378772 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:01.380038 30757 leader_election.cc:341] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
W20260806 20:15:01.380682 30756 leader_election.cc:341] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.381036 30756 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:01.382951 31065 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:01.384166 30915 tablet_bootstrap.cc:492] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:01.397104 30913 tablet_bootstrap.cc:492] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:01.397626 30913 ts_tablet_manager.cc:1397] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.000s
I20260806 20:15:01.400010 30913 raft_consensus.cc:348] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.400677 30913 raft_consensus.cc:374] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.400940 30913 raft_consensus.cc:729] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.402916 30915 tablet_bootstrap.cc:654] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.410483 31280 consensus_queue.cc:1035] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:01.412396 30913 consensus_queue.cc:260] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.430866 30913 ts_tablet_manager.cc:1428] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.033s user 0.004s sys 0.001s
I20260806 20:15:01.431339 30914 tablet_bootstrap.cc:492] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:01.431821 30914 ts_tablet_manager.cc:1397] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.069s user 0.035s sys 0.027s
I20260806 20:15:01.434278 30914 raft_consensus.cc:348] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.434945 30914 raft_consensus.cc:374] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.435181 30914 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.435776 30914 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.436475 30913 tablet_bootstrap.cc:492] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:01.437867 30914 ts_tablet_manager.cc:1428] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 20:15:01.439736 30914 tablet_bootstrap.cc:492] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:01.439817 31232 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.440326 31232 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.442394 31232 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.443085 31088 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.443552 31088 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.443980 30913 tablet_bootstrap.cc:654] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.444679 30914 tablet_bootstrap.cc:654] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.446492 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:01.447903 30676 leader_election.cc:341] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.450664 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:01.451808 30787 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:15:01.453339 30915 tablet_bootstrap.cc:492] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:01.453899 30915 ts_tablet_manager.cc:1397] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.070s user 0.012s sys 0.000s
I20260806 20:15:01.457242 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:01.457926 30865 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:15:01.458039 31065 consensus_queue.cc:1035] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:01.459025 30757 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:01.461024 31233 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:01.461380 31233 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:01.461742 31233 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.466430 30676 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:01.456395 30915 raft_consensus.cc:348] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.470237 30915 raft_consensus.cc:374] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.470592 30915 raft_consensus.cc:729] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.472327 31245 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:01.476442 30915 consensus_queue.cc:260] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.477150 31245 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:01.445304 31088 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.477568 31245 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.480984 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.485764 31233 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:01.489559 30756 leader_election.cc:341] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.492780 30915 ts_tablet_manager.cc:1428] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20260806 20:15:01.493934 30915 tablet_bootstrap.cc:492] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:01.505753 31233 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.506796 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:01.507411 30867 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.521234 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:01.523280 30756 leader_election.cc:341] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.533591 31088 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.534279 31088 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.536155 31088 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.537518 30915 tablet_bootstrap.cc:654] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.538496 30867 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:01.539235 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:01.540103 30788 raft_consensus.cc:2379] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 1: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:01.540977 30757 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:01.540796 31245 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.542356 31233 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:01.542853 31233 raft_consensus.cc:686] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:01.543541 31245 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.541134 30914 tablet_bootstrap.cc:492] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:01.543645 31233 consensus_queue.cc:237] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.544459 30914 ts_tablet_manager.cc:1397] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.105s user 0.031s sys 0.041s
I20260806 20:15:01.544667 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:01.545269 30788 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.546830 30914 raft_consensus.cc:348] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.547561 30914 raft_consensus.cc:374] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.547811 30914 raft_consensus.cc:729] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.548424 30914 consensus_queue.cc:260] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.548864 30913 tablet_bootstrap.cc:492] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:01.549342 30913 ts_tablet_manager.cc:1397] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.113s user 0.022s sys 0.033s
I20260806 20:15:01.550609 30914 ts_tablet_manager.cc:1428] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:15:01.550729 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:01.551633 30914 tablet_bootstrap.cc:492] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
W20260806 20:15:01.552470 30676 leader_election.cc:341] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.552264 30913 raft_consensus.cc:348] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.555858 30913 raft_consensus.cc:374] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.557233 31232 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.557652 30913 raft_consensus.cc:729] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.557834 31232 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.560721 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.561390 30788 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
W20260806 20:15:01.562089 30756 leader_election.cc:341] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.563352 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:01.565277 30757 leader_election.cc:341] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.565728 30757 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:01.566516 30914 tablet_bootstrap.cc:654] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.566917 31280 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:01.567175 30676 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:01.568770 31245 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:01.568830 30913 consensus_queue.cc:260] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.571024 30913 ts_tablet_manager.cc:1428] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.021s user 0.003s sys 0.001s
I20260806 20:15:01.571936 30913 tablet_bootstrap.cc:492] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:01.573742 31245 raft_consensus.cc:686] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:01.572229 31232 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.574647 31245 consensus_queue.cc:237] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.579600 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:01.581252 30676 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:01.582149 31088 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.588382 31088 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:01.606702 30603 catalog_manager.cc:5495] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:01.607648 30915 tablet_bootstrap.cc:492] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:01.608106 30915 ts_tablet_manager.cc:1397] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.114s user 0.015s sys 0.021s
I20260806 20:15:01.616138 30914 tablet_bootstrap.cc:492] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:01.617481 30914 ts_tablet_manager.cc:1397] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.066s user 0.016s sys 0.000s
I20260806 20:15:01.617648 30601 catalog_manager.cc:5495] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:01.619617 30914 raft_consensus.cc:348] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.620189 30914 raft_consensus.cc:374] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.620442 30914 raft_consensus.cc:729] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.620602 30913 tablet_bootstrap.cc:654] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.621191 30914 consensus_queue.cc:260] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.621826 31233 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.622287 31233 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.623646 30914 ts_tablet_manager.cc:1428] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:15:01.624150 31233 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.626482 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:01.627130 30867 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:15:01.625744 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.629696 30757 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:01.630694 31232 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:01.631033 31232 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:01.631336 31232 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.625710 30915 raft_consensus.cc:348] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.632522 30915 raft_consensus.cc:374] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.632859 30915 raft_consensus.cc:729] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.633468 30915 consensus_queue.cc:260] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.634487 30713 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:15:01.635968 30915 ts_tablet_manager.cc:1428] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260806 20:15:01.637300 30915 tablet_bootstrap.cc:492] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:01.637357 30871 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.638564 31254 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.639025 31254 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.641070 31254 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:01.642620 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:01.643260 30789 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:15:01.644336 30835 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:01.645658 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.645955 30995 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:01.646345 30995 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:01.646659 30995 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:15:01.650249 30834 leader_election.cc:341] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.651332 30995 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.653765 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:01.653977 30915 tablet_bootstrap.cc:654] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.653934 30913 tablet_bootstrap.cc:492] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:01.654644 30913 ts_tablet_manager.cc:1397] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.003s
W20260806 20:15:01.656102 30834 leader_election.cc:341] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.657212 30913 raft_consensus.cc:348] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.657982 30913 raft_consensus.cc:374] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.658286 30913 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.659029 30913 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.664285 31254 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:01.670925 30995 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:01.671996 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:01.672664 30788 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.679196 31232 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.683210 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:01.683851 30709 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.692732 30913 ts_tablet_manager.cc:1428] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20260806 20:15:01.693755 30913 tablet_bootstrap.cc:492] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:01.696480 30914 tablet_bootstrap.cc:492] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:01.692466 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:01.700371 30867 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.713279 30788 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:01.713980 31233 consensus_queue.cc:1035] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:01.731999 30915 tablet_bootstrap.cc:492] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:01.720571 30709 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:01.732550 30915 ts_tablet_manager.cc:1397] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.095s user 0.047s sys 0.034s
I20260806 20:15:01.736860 30756 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:01.737849 31280 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:01.738328 31074 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:01.738395 31280 raft_consensus.cc:686] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:01.740523 31280 consensus_queue.cc:237] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.749769 31254 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.746311 30835 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:01.750294 31254 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.751523 30871 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.752301 31254 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:01.754932 30913 tablet_bootstrap.cc:654] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.759292 31233 consensus_queue.cc:1035] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:01.760347 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.761760 31074 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:01.762207 31074 raft_consensus.cc:686] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:01.762934 31074 consensus_queue.cc:237] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.766499 30915 raft_consensus.cc:348] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:01.769901 30834 leader_election.cc:341] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.771030 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:01.785768 30835 leader_election.cc:341] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.786253 30835 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:01.777824 30867 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:01.681202 31232 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.788112 30915 raft_consensus.cc:374] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.788435 30915 raft_consensus.cc:729] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.790091 30914 tablet_bootstrap.cc:654] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.790860 31254 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:01.793203 30915 consensus_queue.cc:260] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:01.803884 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 58554.5us
I20260806 20:15:01.810339 30601 catalog_manager.cc:5495] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:01.812470 30915 ts_tablet_manager.cc:1428] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.080s user 0.009s sys 0.000s
I20260806 20:15:01.813517 30915 tablet_bootstrap.cc:492] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:01.817625 31074 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.818594 30913 tablet_bootstrap.cc:492] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:01.819036 30913 ts_tablet_manager.cc:1397] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.125s user 0.015s sys 0.003s
I20260806 20:15:01.818128 31074 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.835784 30913 raft_consensus.cc:348] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.836901 30913 raft_consensus.cc:374] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.837267 30913 raft_consensus.cc:729] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.838994 31074 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:01.839102 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.839789 30913 consensus_queue.cc:260] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.845135 30913 ts_tablet_manager.cc:1428] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.026s user 0.003s sys 0.003s
W20260806 20:15:01.846484 30834 leader_election.cc:341] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.846908 30913 tablet_bootstrap.cc:492] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:01.850591 30601 catalog_manager.cc:5495] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:01.878438 31065 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.886061 31065 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.888953 31065 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:01.889354 30915 tablet_bootstrap.cc:654] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.899230 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:01.906157 30712 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:15:01.886641 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:01.908774 30787 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:15:01.908802 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:01.910578 30756 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:01.911603 31280 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:01.911957 31280 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:01.912300 31280 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
W20260806 20:15:01.919622 30757 leader_election.cc:341] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.922840 30835 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:01.924196 31254 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:01.924553 31254 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:01.924851 31254 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.937182 30913 tablet_bootstrap.cc:654] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.944254 30914 tablet_bootstrap.cc:492] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:01.954002 30914 ts_tablet_manager.cc:1397] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.258s user 0.016s sys 0.005s
I20260806 20:15:01.956650 30914 raft_consensus.cc:348] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.957394 30914 raft_consensus.cc:374] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:01.963200 30914 raft_consensus.cc:729] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:01.963819 30914 consensus_queue.cc:260] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.966522 30914 ts_tablet_manager.cc:1428] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
W20260806 20:15:01.957566 31314 log.cc:924] Time spent T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260806 20:15:01.969719 30914 tablet_bootstrap.cc:492] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
W20260806 20:15:01.958745 30655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 20:15:01.961740 31318 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:01.977576 31318 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.979434 30914 tablet_bootstrap.cc:654] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:01.980440 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:01.981103 30789 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:15:01.957068 31280 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.984092 30676 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:01.984858 31245 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:01.985215 31245 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:01.985563 31245 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.986696 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:01.987284 30712 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.990108 31254 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.993727 30712 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:01.994885 30756 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:01.995671 31232 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:01.996130 31232 raft_consensus.cc:686] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:01.996630 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:01.998358 30834 leader_election.cc:341] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:01.998472 31232 consensus_queue.cc:237] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.999488 31254 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:02.000208 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:02.000440 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:02.001852 30789 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.004071 31280 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.004249 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:02.003620 30676 leader_election.cc:341] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:02.004798 30871 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:01.992799 31245 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.006978 31245 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.008054 30915 tablet_bootstrap.cc:492] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:02.008586 30915 ts_tablet_manager.cc:1397] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.195s user 0.019s sys 0.010s
I20260806 20:15:02.010495 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:02.011138 30788 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.012276 30871 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:02.012398 30915 raft_consensus.cc:348] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.013129 30915 raft_consensus.cc:374] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.013401 30915 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.014051 30915 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.016106 30789 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:02.017246 30835 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:02.017618 30788 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:02.018472 31254 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:02.019187 31254 raft_consensus.cc:686] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:02.019966 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:02.020125 31254 consensus_queue.cc:237] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:01.997855 31318 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:02.021982 30676 leader_election.cc:341] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:02.024467 30676 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:02.025420 30915 ts_tablet_manager.cc:1428] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260806 20:15:02.025620 31317 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:02.026777 30915 tablet_bootstrap.cc:492] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:02.032441 31317 raft_consensus.cc:686] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:02.038534 30913 tablet_bootstrap.cc:492] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:02.039024 30913 ts_tablet_manager.cc:1397] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.192s user 0.017s sys 0.005s
I20260806 20:15:02.044061 30603 catalog_manager.cc:5495] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:02.048302 30913 raft_consensus.cc:348] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.049022 30913 raft_consensus.cc:374] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.046844 31317 consensus_queue.cc:237] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.049289 30913 raft_consensus.cc:729] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.050019 30913 consensus_queue.cc:260] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.051255 30914 tablet_bootstrap.cc:492] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:02.051760 30914 ts_tablet_manager.cc:1397] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.082s user 0.017s sys 0.007s
I20260806 20:15:02.054423 30913 ts_tablet_manager.cc:1428] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260806 20:15:02.046113 30601 catalog_manager.cc:5495] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:02.061581 30913 tablet_bootstrap.cc:492] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207: Bootstrap starting.
I20260806 20:15:02.067946 31280 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:02.070333 30914 raft_consensus.cc:348] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.071247 30914 raft_consensus.cc:374] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.071570 30914 raft_consensus.cc:729] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.072364 30914 consensus_queue.cc:260] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.074426 30913 tablet_bootstrap.cc:654] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.074892 30914 ts_tablet_manager.cc:1428] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.023s user 0.006s sys 0.002s
I20260806 20:15:02.075856 30914 tablet_bootstrap.cc:492] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
W20260806 20:15:02.082660 30756 consensus_peers.cc:487] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 -> Peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Couldn't send request to peer 42cabca48db7403b8523a3382a046043. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:02.084050 30915 tablet_bootstrap.cc:654] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.082815 30914 tablet_bootstrap.cc:654] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.085294 30913 tablet_bootstrap.cc:492] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207: No bootstrap required, opened a new log
I20260806 20:15:02.085768 30913 ts_tablet_manager.cc:1397] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.006s
I20260806 20:15:02.088591 30913 raft_consensus.cc:348] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.089535 30913 raft_consensus.cc:374] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.089825 30913 raft_consensus.cc:729] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.090803 30913 consensus_queue.cc:260] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.088404 30603 catalog_manager.cc:5495] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:02.099273 30913 ts_tablet_manager.cc:1428] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260806 20:15:02.107085 30914 tablet_bootstrap.cc:492] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:02.107689 30914 ts_tablet_manager.cc:1397] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.001s
I20260806 20:15:02.113242 30914 raft_consensus.cc:348] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.116786 30915 tablet_bootstrap.cc:492] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:02.123044 30915 ts_tablet_manager.cc:1397] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.096s user 0.015s sys 0.011s
I20260806 20:15:02.118837 30914 raft_consensus.cc:374] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.125008 30914 raft_consensus.cc:729] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.125749 30914 consensus_queue.cc:260] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.126010 30915 raft_consensus.cc:348] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.127815 30914 ts_tablet_manager.cc:1428] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20260806 20:15:02.128831 30914 tablet_bootstrap.cc:492] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:02.127858 30915 raft_consensus.cc:374] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.129317 30915 raft_consensus.cc:729] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.130007 30915 consensus_queue.cc:260] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.132232 30915 ts_tablet_manager.cc:1428] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260806 20:15:02.133422 30914 tablet_bootstrap.cc:654] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.151355 30915 tablet_bootstrap.cc:492] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:02.159279 31232 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:02.159838 31232 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.161813 31232 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.164911 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:02.165616 30712 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
W20260806 20:15:02.167632 30676 consensus_peers.cc:487] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 -> Peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Couldn't send request to peer 58d59beb597f4ed99d84d102ddb41ff5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:02.167723 30915 tablet_bootstrap.cc:654] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.168622 30756 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:02.169416 31065 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:02.169893 31065 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.172091 31065 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.172401 30914 tablet_bootstrap.cc:492] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:02.172683 31065 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:02.173046 30914 ts_tablet_manager.cc:1397] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.044s user 0.021s sys 0.017s
I20260806 20:15:02.173098 31065 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:02.175650 30914 raft_consensus.cc:348] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.177281 31317 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:02.181717 31065 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.182775 31317 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.184713 31317 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.185245 31317 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:02.179077 30915 tablet_bootstrap.cc:492] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:02.179355 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:02.185945 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:02.186995 30914 raft_consensus.cc:374] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.186206 30915 ts_tablet_manager.cc:1397] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.012s
I20260806 20:15:02.187309 30914 raft_consensus.cc:729] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
W20260806 20:15:02.187587 30676 leader_election.cc:341] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:02.188002 30914 consensus_queue.cc:260] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:02.188546 30757 leader_election.cc:341] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:02.188973 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:02.190142 30914 ts_tablet_manager.cc:1428] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
W20260806 20:15:02.190495 30756 leader_election.cc:341] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:02.190891 30756 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:02.191047 30914 tablet_bootstrap.cc:492] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:02.188269 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:02.191782 30867 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 0.
I20260806 20:15:02.193740 31311 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:02.197188 30914 tablet_bootstrap.cc:654] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.196733 31065 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.201702 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:02.198009 30915 raft_consensus.cc:348] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.202423 30789 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:15:02.202495 30915 raft_consensus.cc:374] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.204507 30915 raft_consensus.cc:729] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.205299 30915 consensus_queue.cc:260] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.206542 31065 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.209048 30915 ts_tablet_manager.cc:1428] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20260806 20:15:02.209299 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:02.210457 30867 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.211248 30915 tablet_bootstrap.cc:492] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:02.215088 30914 tablet_bootstrap.cc:492] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:02.215595 30914 ts_tablet_manager.cc:1397] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.007s
I20260806 20:15:02.217084 30915 tablet_bootstrap.cc:654] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.218299 30914 raft_consensus.cc:348] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.218986 30914 raft_consensus.cc:374] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.219244 30914 raft_consensus.cc:729] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.220244 31232 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:02.219966 30914 consensus_queue.cc:260] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.221683 31074 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:02.222080 31074 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.223721 31074 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:02.229418 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:02.230944 30676 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:02.231654 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:02.232556 30789 raft_consensus.cc:2379] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 1: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:02.233076 31130 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:02.233480 31130 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:02.233821 31130 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.234869 30914 ts_tablet_manager.cc:1428] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.019s user 0.005s sys 0.003s
I20260806 20:15:02.235985 30914 tablet_bootstrap.cc:492] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5: Bootstrap starting.
I20260806 20:15:02.245010 30834 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:02.246011 30867 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:02.247925 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:02.248668 30707 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.256121 30707 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:02.257360 30914 tablet_bootstrap.cc:654] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.259434 30915 tablet_bootstrap.cc:492] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:02.261368 30915 ts_tablet_manager.cc:1397] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.050s user 0.033s sys 0.013s
I20260806 20:15:02.263579 31317 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:02.264029 31317 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.266103 31317 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.267592 31254 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.267033 30915 raft_consensus.cc:348] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.268944 30915 raft_consensus.cc:374] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.269223 30915 raft_consensus.cc:729] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.269819 30915 consensus_queue.cc:260] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.272248 30915 ts_tablet_manager.cc:1428] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
I20260806 20:15:02.273487 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:02.274081 30915 tablet_bootstrap.cc:492] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:02.274222 30867 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:15:02.275032 31254 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:02.275409 30676 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:02.276297 31317 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:02.276713 31317 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:02.277082 31241 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:02.277626 31241 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.279613 31241 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:02.281054 30915 tablet_bootstrap.cc:654] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.277099 31317 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.283233 30756 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:02.291368 31311 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:02.291905 31311 raft_consensus.cc:686] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:02.293390 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:02.294083 30790 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 0.
I20260806 20:15:02.292734 31311 consensus_queue.cc:237] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.295734 30835 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:02.296401 31337 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:15:02.296738 31337 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:02.297014 31337 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.305708 31337 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.304527 31317 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.307750 31337 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:02.308755 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:02.309556 30788 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.309775 30914 tablet_bootstrap.cc:492] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5: No bootstrap required, opened a new log
I20260806 20:15:02.310222 30914 ts_tablet_manager.cc:1397] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.000s
I20260806 20:15:02.311884 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:02.312538 30790 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 0.
I20260806 20:15:02.313465 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:02.313990 30790 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.318274 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:02.256388 31130 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.319502 31317 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.320236 30790 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:02.320931 31130 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.321528 30788 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:02.322525 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:02.323663 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:02.323513 30914 raft_consensus.cc:348] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.323277 31337 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:02.324579 30914 raft_consensus.cc:374] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.323099 30867 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.324728 31337 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.327215 31337 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:02.328136 30835 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:02.328882 31337 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:02.329612 31337 raft_consensus.cc:686] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:02.329847 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:02.331010 31337 consensus_queue.cc:237] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:02.333638 30834 leader_election.cc:341] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:02.351063 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:02.351706 30790 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.355087 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:02.357846 30676 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:02.359354 31130 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:02.359843 31130 raft_consensus.cc:686] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:02.360674 31130 consensus_queue.cc:237] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.364169 30790 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:02.363703 30603 catalog_manager.cc:5495] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:02.362053 30712 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.324913 30914 raft_consensus.cc:729] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.366940 30915 tablet_bootstrap.cc:492] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:02.367476 30915 ts_tablet_manager.cc:1397] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.094s user 0.020s sys 0.031s
I20260806 20:15:02.367914 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:02.370337 30915 raft_consensus.cc:348] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.371232 30915 raft_consensus.cc:374] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.371701 30915 raft_consensus.cc:729] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.372479 30915 consensus_queue.cc:260] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.374145 30915 ts_tablet_manager.cc:1428] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:15:02.340441 30867 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:02.376247 30915 tablet_bootstrap.cc:492] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043: Bootstrap starting.
I20260806 20:15:02.370781 30914 consensus_queue.cc:260] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.379271 30676 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:02.380290 30869 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.380306 31317 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:15:02.380980 31317 raft_consensus.cc:686] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:02.381940 31317 consensus_queue.cc:237] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.391813 30914 ts_tablet_manager.cc:1428] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent starting tablet: real 0.081s user 0.006s sys 0.000s
I20260806 20:15:02.395690 31254 consensus_queue.cc:1035] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260806 20:15:02.390460 30869 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:02.402911 31311 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260806 20:15:02.410970 30834 leader_election.cc:341] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 20:15:02.412271 30834 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:02.412979 31074 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:15:02.413530 31074 consensus_queue.cc:1035] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:02.419342 31088 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:02.419854 31088 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.421624 31088 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.425513 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:02.428443 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:02.429342 30790 raft_consensus.cc:2379] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 1: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:02.433372 30915 tablet_bootstrap.cc:654] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043: Neither blocks nor log segments found. Creating new log.
I20260806 20:15:02.443342 30601 catalog_manager.cc:5495] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:02.441340 30602 catalog_manager.cc:5495] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:02.473907 31318 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:02.475513 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 45541.8us
W20260806 20:15:02.477111 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 63784.5us
I20260806 20:15:02.486013 30676 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:02.489434 31130 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.474454 31318 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.499672 31318 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:02.503654 31130 raft_consensus.cc:2726] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 20:15:02.548619 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 62472us
I20260806 20:15:02.551551 30915 tablet_bootstrap.cc:492] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043: No bootstrap required, opened a new log
I20260806 20:15:02.552747 30915 ts_tablet_manager.cc:1397] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043: Time spent bootstrapping tablet: real 0.177s user 0.013s sys 0.000s
W20260806 20:15:02.568550 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 70472.8us
I20260806 20:15:02.571087 31233 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:02.568198 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:02.582341 31346 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:02.585114 31130 consensus_queue.cc:1035] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:02.596153 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:02.650594 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:02.652848 30915 raft_consensus.cc:348] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.653951 30915 raft_consensus.cc:374] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:15:02.654261 30915 raft_consensus.cc:729] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Initialized, Role: FOLLOWER
I20260806 20:15:02.659590 31347 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:02.660230 30915 consensus_queue.cc:260] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:02.665468 30915 ts_tablet_manager.cc:1428] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043: Time spent starting tablet: real 0.112s user 0.010s sys 0.002s
I20260806 20:15:02.675261 30865 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.686205 30712 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.688766 31347 consensus_queue.cc:1035] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:02.718675 31233 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:02.729626 30602 catalog_manager.cc:5495] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
W20260806 20:15:02.736030 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 77153.9us
I20260806 20:15:02.785959 31347 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 20:15:02.808348 31254 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:02.839488 31317 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:02.875231 30713 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:15:02.883368 31233 consensus_queue.cc:1035] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:02.883523 31318 consensus_queue.cc:1035] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:02.969707 31254 consensus_queue.cc:1035] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 20:15:02.989758 31065 consensus_queue.cc:1035] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 20:15:03.005538 31245 consensus_queue.cc:1035] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260806 20:15:03.013864 31346 consensus_queue.cc:1035] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:15:04.281888 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:15:04.292249 31377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:15:04.301286 31378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:15:04.335788 31380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:15:04.336375 29419 server_base.cc:1034] running on GCE node
I20260806 20:15:04.337474 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 20:15:04.337769 29419 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 20:15:04.337944 29419 hybrid_clock.cc:648] HybridClock initialized: now 1786047304337932 us; error 0 us; skew 500 ppm
I20260806 20:15:04.338512 29419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:15:04.342391 29419 webserver.cc:460] Webserver started at http://127.28.186.196:41215/ using document root <none> and password file <none>
I20260806 20:15:04.343209 29419 fs_manager.cc:359] Metadata directory not provided
I20260806 20:15:04.343571 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:15:04.344092 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:15:04.345866 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-3-root/instance:
uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
format_stamp: "Formatted at 2026-08-06 20:15:04 on dist-test-slave-49h3"
I20260806 20:15:04.351895 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260806 20:15:04.356031 31385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:15:04.357153 29419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:15:04.357585 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-3-root
uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
format_stamp: "Formatted at 2026-08-06 20:15:04 on dist-test-slave-49h3"
I20260806 20:15:04.357993 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:15:04.375628 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:15:04.377138 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:15:04.378906 29419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:15:04.381512 29419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:15:04.381716 29419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:15:04.381947 29419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:15:04.382097 29419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:15:04.451933 29419 rpc_server.cc:307] RPC server started. Bound to: 127.28.186.196:35519
I20260806 20:15:04.452049 31449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.186.196:35519 every 8 connection(s)
I20260806 20:15:04.456094 29419 auto_leader_rebalancer-test.cc:413] add a tserver: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:04.470644 31450 heartbeater.cc:347] Connected to a master server at 127.28.186.254:40443
I20260806 20:15:04.471067 31450 heartbeater.cc:464] Registering TS with master...
I20260806 20:15:04.471755 31450 heartbeater.cc:511] Master 127.28.186.254:40443 requested a full tablet report, sending...
I20260806 20:15:04.473564 30603 ts_manager.cc:194] Registered new tserver with Master: 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:04.475498 30603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34006
I20260806 20:15:05.271518 30788 consensus_queue.cc:237] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:05.282193 30869 raft_consensus.cc:1261] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:05.283805 31376 consensus_queue.cc:1035] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:05.286659 30710 raft_consensus.cc:1261] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:05.288218 31459 consensus_queue.cc:1035] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:15:05.302860 30754 consensus_peers.cc:487] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 616aae135c70498da00a9e8a1a61b1cd. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:05.306130 31459 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.308432 30710 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.316556 30869 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.327390 30603 catalog_manager.cc:5495] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:05.355248 30710 consensus_queue.cc:237] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:05.366421 30869 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:05.373776 30788 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:05.380590 31455 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:05.385262 31317 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:05.395850 31317 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.396837 30869 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.435209 30790 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.435793 30710 consensus_queue.cc:237] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:05.444003 30869 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:05.445897 31347 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:05.453536 30601 catalog_manager.cc:5495] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 20:15:05.457517 30673 consensus_peers.cc:487] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1485b55849242feae0e8b2b3b28307f. This is attempt 1: this message will repeat every 5th retry.
W20260806 20:15:05.458587 30673 consensus_peers.cc:487] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f92e9dc9db864f0b8e238a39be31b12e. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:05.462503 30788 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:05.464886 31455 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:05.476249 31317 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.478461 31450 heartbeater.cc:503] Master 127.28.186.254:40443 was elected leader, sending a full tablet report...
I20260806 20:15:05.482088 30865 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.495332 30788 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.500547 30601 catalog_manager.cc:5495] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.502233 30710 consensus_queue.cc:237] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:05.512728 30673 consensus_peers.cc:487] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f3393f94b2649378cb19b57fecfb21e. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:05.513250 30866 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:05.514470 30788 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:05.516016 31317 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:05.517108 31477 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:05.534308 31317 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.535972 30790 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.540812 30866 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:05.552331 30602 catalog_manager.cc:5495] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:05.690402 31482 ts_tablet_manager.cc:927] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:05.693652 31482 tablet_copy_client.cc:316] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.194:42029
I20260806 20:15:05.713508 30804 tablet_copy_service.cc:140] P d4181d4d907f477f9feddc44ba3b9207: Received BeginTabletCopySession request for tablet 616aae135c70498da00a9e8a1a61b1cd from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:53732)
I20260806 20:15:05.714190 30804 tablet_copy_service.cc:161] P d4181d4d907f477f9feddc44ba3b9207: Beginning new tablet copy session on tablet 616aae135c70498da00a9e8a1a61b1cd from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:53732: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-616aae135c70498da00a9e8a1a61b1cd
I20260806 20:15:05.722324 30804 tablet_copy_source_session.cc:205] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:05.729790 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616aae135c70498da00a9e8a1a61b1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:05.747207 31482 tablet_copy_client.cc:791] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:05.748121 31482 tablet_copy_client.cc:655] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:05.756313 31482 tablet_copy_client.cc:531] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:05.766637 31482 tablet_bootstrap.cc:492] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:05.789533 31482 tablet_bootstrap.cc:492] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:05.790391 31482 tablet_bootstrap.cc:492] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:05.791035 31482 ts_tablet_manager.cc:1397] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.001s
I20260806 20:15:05.794382 31482 raft_consensus.cc:348] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:05.795279 31482 raft_consensus.cc:729] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:05.795950 31482 consensus_queue.cc:260] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:05.800236 31482 ts_tablet_manager.cc:1428] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20260806 20:15:05.805315 30804 tablet_copy_service.cc:342] P d4181d4d907f477f9feddc44ba3b9207: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-616aae135c70498da00a9e8a1a61b1cd received from {username='slave'} at 127.0.0.1:53732
I20260806 20:15:05.805817 30804 tablet_copy_service.cc:433] P d4181d4d907f477f9feddc44ba3b9207: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-616aae135c70498da00a9e8a1a61b1cd on tablet 616aae135c70498da00a9e8a1a61b1cd with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:05.822297 31482 ts_tablet_manager.cc:927] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:05.823745 31482 tablet_copy_client.cc:316] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:05.835999 30724 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet e1485b55849242feae0e8b2b3b28307f from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:05.836570 30724 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet e1485b55849242feae0e8b2b3b28307f from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-e1485b55849242feae0e8b2b3b28307f
I20260806 20:15:05.843768 30724 tablet_copy_source_session.cc:205] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:05.846989 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1485b55849242feae0e8b2b3b28307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:05.860713 31482 tablet_copy_client.cc:791] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:05.861253 31482 tablet_copy_client.cc:655] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:05.864634 31482 tablet_copy_client.cc:531] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:05.876464 31482 tablet_bootstrap.cc:492] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:05.906559 31482 tablet_bootstrap.cc:492] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:05.907482 31482 tablet_bootstrap.cc:492] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:05.908237 31482 ts_tablet_manager.cc:1397] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.016s
I20260806 20:15:05.910604 31482 raft_consensus.cc:348] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:05.911442 31482 raft_consensus.cc:729] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:05.912061 31482 consensus_queue.cc:260] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:05.914251 31482 ts_tablet_manager.cc:1428] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 20:15:05.915769 30724 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-e1485b55849242feae0e8b2b3b28307f received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:05.916165 30724 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-e1485b55849242feae0e8b2b3b28307f on tablet e1485b55849242feae0e8b2b3b28307f with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:05.980937 31482 ts_tablet_manager.cc:927] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:05.982388 31482 tablet_copy_client.cc:316] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:05.983906 30724 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet f92e9dc9db864f0b8e238a39be31b12e from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:05.984380 30724 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet f92e9dc9db864f0b8e238a39be31b12e from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-f92e9dc9db864f0b8e238a39be31b12e
I20260806 20:15:05.990397 30724 tablet_copy_source_session.cc:205] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:05.992810 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f92e9dc9db864f0b8e238a39be31b12e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:06.006731 31482 tablet_copy_client.cc:791] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:06.007490 31482 tablet_copy_client.cc:655] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:06.012820 31482 tablet_copy_client.cc:531] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:06.021488 31482 tablet_bootstrap.cc:492] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:06.043500 31482 tablet_bootstrap.cc:492] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:06.044175 31482 tablet_bootstrap.cc:492] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:06.044672 31482 ts_tablet_manager.cc:1397] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.003s
I20260806 20:15:06.046651 31482 raft_consensus.cc:348] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:06.047339 31482 raft_consensus.cc:729] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:06.047792 31482 consensus_queue.cc:260] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:06.049208 31482 ts_tablet_manager.cc:1428] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260806 20:15:06.053234 30724 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-f92e9dc9db864f0b8e238a39be31b12e received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:06.053663 30724 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-f92e9dc9db864f0b8e238a39be31b12e on tablet f92e9dc9db864f0b8e238a39be31b12e with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:06.082947 31422 raft_consensus.cc:1203] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:06.092073 31482 ts_tablet_manager.cc:927] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:06.094429 31482 tablet_copy_client.cc:316] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:06.095947 30724 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet 0f3393f94b2649378cb19b57fecfb21e from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:06.096364 30724 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet 0f3393f94b2649378cb19b57fecfb21e from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-0f3393f94b2649378cb19b57fecfb21e
I20260806 20:15:06.103920 30724 tablet_copy_source_session.cc:205] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:06.106789 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f3393f94b2649378cb19b57fecfb21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:06.119117 31482 tablet_copy_client.cc:791] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:06.119730 31482 tablet_copy_client.cc:655] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:06.123172 31482 tablet_copy_client.cc:531] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:06.131181 31482 tablet_bootstrap.cc:492] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:06.155821 31482 tablet_bootstrap.cc:492] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:06.156814 31482 tablet_bootstrap.cc:492] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:06.157536 31482 ts_tablet_manager.cc:1397] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.012s
I20260806 20:15:06.159941 31482 raft_consensus.cc:348] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:06.160733 31482 raft_consensus.cc:729] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:06.161327 31482 consensus_queue.cc:260] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:06.165743 31482 ts_tablet_manager.cc:1428] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 20:15:06.167444 30724 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-0f3393f94b2649378cb19b57fecfb21e received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:06.167855 30724 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-0f3393f94b2649378cb19b57fecfb21e on tablet 0f3393f94b2649378cb19b57fecfb21e with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:06.222137 31422 raft_consensus.cc:1203] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:06.502905 31422 raft_consensus.cc:1203] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:06.558758 31493 raft_consensus.cc:1050] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:06.561652 31493 consensus_queue.cc:237] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:06.569170 31422 raft_consensus.cc:1261] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:06.570559 31311 consensus_queue.cc:1035] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:06.571440 30869 raft_consensus.cc:1261] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:06.571930 30713 raft_consensus.cc:1261] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:06.573271 31493 consensus_queue.cc:1035] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:06.575412 31311 consensus_queue.cc:1035] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:06.590749 31423 raft_consensus.cc:1203] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:06.608497 31459 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.611096 31422 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.619393 30866 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.629628 30711 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.630184 30601 catalog_manager.cc:5495] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.669902 30790 consensus_queue.cc:237] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:06.676434 31422 raft_consensus.cc:1261] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:06.676800 30866 raft_consensus.cc:1261] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:06.678208 31501 consensus_queue.cc:1035] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:06.679252 31502 consensus_queue.cc:1035] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:06.689415 31501 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.691815 31422 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.699306 30866 raft_consensus.cc:2932] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.704262 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 616aae135c70498da00a9e8a1a61b1cd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:06.715163 30602 catalog_manager.cc:5495] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.731281 30700 tablet_service.cc:1514] Processing DeleteTablet for tablet 616aae135c70498da00a9e8a1a61b1cd with delete_type TABLET_DATA_TOMBSTONED (TS 42cabca48db7403b8523a3382a046043 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20260806 20:15:06.735149 31507 tablet_replica.cc:330] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043: stopping tablet replica
I20260806 20:15:06.736159 31507 raft_consensus.cc:2229] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:06.736917 31507 raft_consensus.cc:2258] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:06.743628 31507 ts_tablet_manager.cc:1905] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:06.758081 31507 ts_tablet_manager.cc:1918] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:06.758460 31507 log.cc:1196] T 616aae135c70498da00a9e8a1a61b1cd P 42cabca48db7403b8523a3382a046043: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/wals/616aae135c70498da00a9e8a1a61b1cd
I20260806 20:15:06.760030 30589 catalog_manager.cc:4841] TS 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): tablet 616aae135c70498da00a9e8a1a61b1cd (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:06.791723 31477 raft_consensus.cc:1050] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:06.794086 31477 consensus_queue.cc:237] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:06.804996 30790 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:06.804941 30866 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:06.806519 31455 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:06.808843 31422 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:06.810909 31317 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:06.813104 31455 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:06.824646 31477 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.826455 31422 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.841049 30787 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.844177 30600 catalog_manager.cc:5495] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.848112 30869 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.872011 30713 consensus_queue.cc:237] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:06.878255 31422 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:06.880041 31511 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:06.888381 30790 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:06.889309 31455 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.892077 31477 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:06.890618 31423 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.898357 30787 raft_consensus.cc:2932] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.913303 30602 catalog_manager.cc:5495] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:06.916018 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e1485b55849242feae0e8b2b3b28307f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:06.924358 30855 tablet_service.cc:1514] Processing DeleteTablet for tablet e1485b55849242feae0e8b2b3b28307f with delete_type TABLET_DATA_TOMBSTONED (TS 58d59beb597f4ed99d84d102ddb41ff5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60882
I20260806 20:15:06.930393 31514 tablet_replica.cc:330] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5: stopping tablet replica
I20260806 20:15:06.931582 31514 raft_consensus.cc:2229] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:06.932314 31514 raft_consensus.cc:2258] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:06.942077 31514 ts_tablet_manager.cc:1905] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:06.955926 31514 ts_tablet_manager.cc:1918] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:06.956331 31514 log.cc:1196] T e1485b55849242feae0e8b2b3b28307f P 58d59beb597f4ed99d84d102ddb41ff5: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/wals/e1485b55849242feae0e8b2b3b28307f
I20260806 20:15:06.957922 30590 catalog_manager.cc:4841] TS 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): tablet e1485b55849242feae0e8b2b3b28307f (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:06.985134 31455 raft_consensus.cc:1050] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:06.987375 31455 consensus_queue.cc:237] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:06.995157 30866 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:06.995289 30790 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:06.995635 31423 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:06.997030 31477 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:06.998603 31455 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:06.999600 31511 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:07.015285 31455 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.017190 30866 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.024955 30790 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.026458 31423 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.037055 30600 catalog_manager.cc:5495] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:07.047870 31511 raft_consensus.cc:1050] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:07.050549 31511 consensus_queue.cc:237] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:07.060969 30866 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:07.062958 31455 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:07.063303 30713 consensus_queue.cc:237] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:07.066175 31422 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:07.070324 31317 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:07.074992 30867 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:07.077956 30790 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:07.081130 31317 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:07.081318 31511 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 20:15:07.080917 31423 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:07.090471 31511 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:07.095780 31317 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.097795 30865 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.109756 31455 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.114509 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0f3393f94b2649378cb19b57fecfb21e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:07.114500 30602 catalog_manager.cc:5495] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.118707 31422 raft_consensus.cc:2932] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.114946 31423 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.135854 30787 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.136656 30865 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.144907 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f3393f94b2649378cb19b57fecfb21e with delete_type TABLET_DATA_TOMBSTONED (TS d4181d4d907f477f9feddc44ba3b9207 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33214
I20260806 20:15:07.158725 30601 catalog_manager.cc:5495] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.166806 31524 tablet_replica.cc:330] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207: stopping tablet replica
I20260806 20:15:07.167913 31524 raft_consensus.cc:2229] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:07.168545 31524 raft_consensus.cc:2258] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:07.236311 31524 ts_tablet_manager.cc:1905] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:07.257179 30710 consensus_queue.cc:237] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:07.274518 30866 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:07.262422 31524 ts_tablet_manager.cc:1918] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:07.277092 31511 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:07.277678 31524 log.cc:1196] T 0f3393f94b2649378cb19b57fecfb21e P d4181d4d907f477f9feddc44ba3b9207: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/wals/0f3393f94b2649378cb19b57fecfb21e
I20260806 20:15:07.281952 31423 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:07.283208 30590 catalog_manager.cc:4841] TS d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): tablet 0f3393f94b2649378cb19b57fecfb21e (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:07.286832 31455 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:07.293592 31455 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.297770 31423 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.295584 30867 raft_consensus.cc:2932] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.305433 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f92e9dc9db864f0b8e238a39be31b12e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:07.318311 30601 catalog_manager.cc:5495] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:07.326980 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet f92e9dc9db864f0b8e238a39be31b12e with delete_type TABLET_DATA_TOMBSTONED (TS d4181d4d907f477f9feddc44ba3b9207 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33214
I20260806 20:15:07.327896 31524 tablet_replica.cc:330] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207: stopping tablet replica
I20260806 20:15:07.329007 31524 raft_consensus.cc:2229] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:07.329530 31524 raft_consensus.cc:2258] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:07.332492 31524 ts_tablet_manager.cc:1905] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:07.343490 31524 ts_tablet_manager.cc:1918] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:07.343813 31524 log.cc:1196] T f92e9dc9db864f0b8e238a39be31b12e P d4181d4d907f477f9feddc44ba3b9207: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/wals/f92e9dc9db864f0b8e238a39be31b12e
I20260806 20:15:07.345207 30590 catalog_manager.cc:4841] TS d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): tablet f92e9dc9db864f0b8e238a39be31b12e (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:08.404259 30866 consensus_queue.cc:237] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:08.410507 30710 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:08.411440 30790 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:08.412043 31454 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:08.413071 31525 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:15:08.431129 30832 consensus_peers.cc:487] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01fd0995e74a449090ea2b00062f625e. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:08.433073 31489 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.437984 30710 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.449154 30790 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.456202 30601 catalog_manager.cc:5495] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.459630 30790 consensus_queue.cc:237] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:08.470604 30754 consensus_peers.cc:487] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79dfdb4a0e9a40d39c83f2c1fd1e777b. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:08.470613 30866 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:08.472921 30710 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:08.474648 31526 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:08.475589 31502 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:08.488469 31526 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.491099 30711 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.491792 30866 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.510833 30865 consensus_queue.cc:237] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:08.513244 30600 catalog_manager.cc:5495] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
W20260806 20:15:08.521579 30832 consensus_peers.cc:487] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0607668d3924d5cba6acd89b2cdc25f. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:08.525240 30790 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:08.527130 31454 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:08.527912 30711 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:08.529253 31525 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:08.536159 31525 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.537905 30790 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.538599 30711 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.551891 30865 consensus_queue.cc:237] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:08.560058 30600 catalog_manager.cc:5495] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.567149 30787 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:08.568471 31525 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:08.569279 30711 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:15:08.569386 30832 consensus_peers.cc:487] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c541d91c0c4487c9a76c6fc6f81a95a. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:08.570600 31525 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:08.577515 31489 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.581336 30711 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.587442 30787 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:08.603223 30602 catalog_manager.cc:5495] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:08.902947 31549 ts_tablet_manager.cc:927] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:08.905175 31549 tablet_copy_client.cc:316] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.195:41735
I20260806 20:15:08.920384 30882 tablet_copy_service.cc:140] P 58d59beb597f4ed99d84d102ddb41ff5: Received BeginTabletCopySession request for tablet e0607668d3924d5cba6acd89b2cdc25f from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:45134)
I20260806 20:15:08.920971 30882 tablet_copy_service.cc:161] P 58d59beb597f4ed99d84d102ddb41ff5: Beginning new tablet copy session on tablet e0607668d3924d5cba6acd89b2cdc25f from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:45134: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-e0607668d3924d5cba6acd89b2cdc25f
I20260806 20:15:08.930411 30882 tablet_copy_source_session.cc:205] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:08.933810 31549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0607668d3924d5cba6acd89b2cdc25f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:08.951928 31549 tablet_copy_client.cc:791] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:08.952557 31549 tablet_copy_client.cc:655] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:08.958053 31549 tablet_copy_client.cc:531] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:08.969084 31549 tablet_bootstrap.cc:492] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:08.976126 31553 ts_tablet_manager.cc:927] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:08.996939 31553 tablet_copy_client.cc:316] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.194:42029
I20260806 20:15:08.999114 30804 tablet_copy_service.cc:140] P d4181d4d907f477f9feddc44ba3b9207: Received BeginTabletCopySession request for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:53732)
I20260806 20:15:08.999683 30804 tablet_copy_service.cc:161] P d4181d4d907f477f9feddc44ba3b9207: Beginning new tablet copy session on tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:53732: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-79dfdb4a0e9a40d39c83f2c1fd1e777b
I20260806 20:15:09.011297 31549 tablet_bootstrap.cc:492] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:09.012269 31549 tablet_bootstrap.cc:492] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:09.012432 30804 tablet_copy_source_session.cc:205] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:09.013902 31549 ts_tablet_manager.cc:1397] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.046s user 0.023s sys 0.005s
I20260806 20:15:09.015647 31553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:09.016700 31549 raft_consensus.cc:348] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:09.017421 31549 raft_consensus.cc:729] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:09.018080 31549 consensus_queue.cc:260] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:09.022394 31549 ts_tablet_manager.cc:1428] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260806 20:15:09.025679 30882 tablet_copy_service.cc:342] P 58d59beb597f4ed99d84d102ddb41ff5: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-e0607668d3924d5cba6acd89b2cdc25f received from {username='slave'} at 127.0.0.1:45134
I20260806 20:15:09.026113 30882 tablet_copy_service.cc:433] P 58d59beb597f4ed99d84d102ddb41ff5: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-e0607668d3924d5cba6acd89b2cdc25f on tablet e0607668d3924d5cba6acd89b2cdc25f with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:09.032291 31549 ts_tablet_manager.cc:927] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:09.040561 31549 tablet_copy_client.cc:316] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.195:41735
I20260806 20:15:09.041889 30882 tablet_copy_service.cc:140] P 58d59beb597f4ed99d84d102ddb41ff5: Received BeginTabletCopySession request for tablet 01fd0995e74a449090ea2b00062f625e from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:45134)
I20260806 20:15:09.042380 30882 tablet_copy_service.cc:161] P 58d59beb597f4ed99d84d102ddb41ff5: Beginning new tablet copy session on tablet 01fd0995e74a449090ea2b00062f625e from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:45134: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-01fd0995e74a449090ea2b00062f625e
I20260806 20:15:09.042935 31553 tablet_copy_client.cc:791] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:09.043543 31553 tablet_copy_client.cc:655] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:09.048381 31553 tablet_copy_client.cc:531] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:09.049362 30882 tablet_copy_source_session.cc:205] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:09.052520 31549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fd0995e74a449090ea2b00062f625e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:09.058517 31553 tablet_bootstrap.cc:492] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:09.066304 31549 tablet_copy_client.cc:791] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:09.066946 31549 tablet_copy_client.cc:655] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:09.071452 31549 tablet_copy_client.cc:531] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:09.079723 31549 tablet_bootstrap.cc:492] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:09.084637 31553 tablet_bootstrap.cc:492] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:09.085546 31553 tablet_bootstrap.cc:492] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:09.086222 31553 ts_tablet_manager.cc:1397] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.012s
I20260806 20:15:09.088961 31553 raft_consensus.cc:348] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:09.089949 31553 raft_consensus.cc:729] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:09.090591 31553 consensus_queue.cc:260] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:09.092905 31553 ts_tablet_manager.cc:1428] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260806 20:15:09.095184 30804 tablet_copy_service.cc:342] P d4181d4d907f477f9feddc44ba3b9207: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-79dfdb4a0e9a40d39c83f2c1fd1e777b received from {username='slave'} at 127.0.0.1:53732
I20260806 20:15:09.095582 30804 tablet_copy_service.cc:433] P d4181d4d907f477f9feddc44ba3b9207: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-79dfdb4a0e9a40d39c83f2c1fd1e777b on tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:09.112601 31549 tablet_bootstrap.cc:492] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:09.113639 31549 tablet_bootstrap.cc:492] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:09.114347 31549 ts_tablet_manager.cc:1397] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.009s
I20260806 20:15:09.116676 31549 raft_consensus.cc:348] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:09.117498 31549 raft_consensus.cc:729] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:09.118047 31549 consensus_queue.cc:260] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:09.119851 31553 ts_tablet_manager.cc:927] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:09.120579 31549 ts_tablet_manager.cc:1428] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 20:15:09.123811 30882 tablet_copy_service.cc:342] P 58d59beb597f4ed99d84d102ddb41ff5: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-01fd0995e74a449090ea2b00062f625e received from {username='slave'} at 127.0.0.1:45134
I20260806 20:15:09.124226 30882 tablet_copy_service.cc:433] P 58d59beb597f4ed99d84d102ddb41ff5: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-01fd0995e74a449090ea2b00062f625e on tablet 01fd0995e74a449090ea2b00062f625e with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:09.134853 31553 tablet_copy_client.cc:316] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.195:41735
I20260806 20:15:09.136312 30882 tablet_copy_service.cc:140] P 58d59beb597f4ed99d84d102ddb41ff5: Received BeginTabletCopySession request for tablet 7c541d91c0c4487c9a76c6fc6f81a95a from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:45134)
I20260806 20:15:09.136813 30882 tablet_copy_service.cc:161] P 58d59beb597f4ed99d84d102ddb41ff5: Beginning new tablet copy session on tablet 7c541d91c0c4487c9a76c6fc6f81a95a from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:45134: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-7c541d91c0c4487c9a76c6fc6f81a95a
I20260806 20:15:09.144205 30882 tablet_copy_source_session.cc:205] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:09.147671 31553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c541d91c0c4487c9a76c6fc6f81a95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:09.163636 31553 tablet_copy_client.cc:791] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:09.164240 31553 tablet_copy_client.cc:655] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:09.168152 31553 tablet_copy_client.cc:531] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:09.177160 31553 tablet_bootstrap.cc:492] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:09.193807 31553 tablet_bootstrap.cc:492] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:09.194736 31553 tablet_bootstrap.cc:492] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:09.195286 31553 ts_tablet_manager.cc:1397] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.005s
I20260806 20:15:09.197988 31553 raft_consensus.cc:348] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:09.198905 31553 raft_consensus.cc:729] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:09.199538 31553 consensus_queue.cc:260] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:09.201941 31553 ts_tablet_manager.cc:1428] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 20:15:09.203409 30882 tablet_copy_service.cc:342] P 58d59beb597f4ed99d84d102ddb41ff5: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-7c541d91c0c4487c9a76c6fc6f81a95a received from {username='slave'} at 127.0.0.1:45134
I20260806 20:15:09.203799 30882 tablet_copy_service.cc:433] P 58d59beb597f4ed99d84d102ddb41ff5: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-7c541d91c0c4487c9a76c6fc6f81a95a on tablet 7c541d91c0c4487c9a76c6fc6f81a95a with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:09.366063 31423 raft_consensus.cc:1203] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:09.482519 31423 raft_consensus.cc:1203] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:09.497687 31423 raft_consensus.cc:1203] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:09.708928 31423 raft_consensus.cc:1203] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:09.821511 31526 raft_consensus.cc:1050] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:09.823663 31526 consensus_queue.cc:237] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:09.830418 31423 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:09.830852 30866 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:09.832305 31501 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:09.832800 30711 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:09.833266 31526 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:09.834137 31564 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:09.855249 31501 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:09.859378 30867 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:09.864566 30711 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:09.869165 31423 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:09.883307 30602 catalog_manager.cc:5495] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:09.920269 30790 consensus_queue.cc:237] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:09.925081 30711 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:09.925107 31423 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:09.926512 31564 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:09.927429 31501 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:09.933398 31501 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:09.942242 30711 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:09.943372 31423 raft_consensus.cc:2932] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:09.956852 30603 catalog_manager.cc:5495] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:09.952373 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:09.971740 30855 tablet_service.cc:1514] Processing DeleteTablet for tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b with delete_type TABLET_DATA_TOMBSTONED (TS 58d59beb597f4ed99d84d102ddb41ff5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60882
I20260806 20:15:09.981906 31578 tablet_replica.cc:330] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5: stopping tablet replica
I20260806 20:15:09.983047 31578 raft_consensus.cc:2229] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:09.983806 31578 raft_consensus.cc:2258] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:09.987293 31578 ts_tablet_manager.cc:1905] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:10.003914 31578 ts_tablet_manager.cc:1918] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:10.004302 31578 log.cc:1196] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58d59beb597f4ed99d84d102ddb41ff5: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/wals/79dfdb4a0e9a40d39c83f2c1fd1e777b
I20260806 20:15:10.005904 30590 catalog_manager.cc:4841] TS 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): tablet 79dfdb4a0e9a40d39c83f2c1fd1e777b (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:10.015380 31489 raft_consensus.cc:1050] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:10.017403 31489 consensus_queue.cc:237] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:10.024317 31565 raft_consensus.cc:1050] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:10.026641 31565 consensus_queue.cc:237] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:10.039498 31422 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:10.040405 30789 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:10.041538 31567 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:10.042440 30713 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:10.044999 31489 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:10.048655 31489 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:10.049621 31423 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:10.051424 31567 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:10.058439 30787 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:10.062790 31525 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:10.064913 30713 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:10.072276 31567 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:10.085783 31567 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.087728 31422 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.096051 31582 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.097396 30789 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.108865 30601 catalog_manager.cc:5495] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.110733 31422 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.117987 30713 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.124641 30600 catalog_manager.cc:5495] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.128007 30711 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.127825 30790 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.169288 30866 consensus_queue.cc:237] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:10.169792 30867 consensus_queue.cc:237] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:10.177227 31421 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:10.178798 31565 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:10.179296 30790 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:10.177268 31422 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:10.180662 30711 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:10.180979 31582 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:10.181571 31525 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:10.182278 31580 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:10.192198 31567 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.193517 31422 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.194823 30711 raft_consensus.cc:2932] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.198007 31489 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.203336 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e0607668d3924d5cba6acd89b2cdc25f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:10.202708 31421 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.205720 31525 raft_consensus.cc:1050] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:10.207886 31525 consensus_queue.cc:237] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:10.208379 30787 raft_consensus.cc:2932] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.217236 30600 catalog_manager.cc:5495] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:10.222337 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 01fd0995e74a449090ea2b00062f625e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:10.228152 30601 catalog_manager.cc:5495] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.240842 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet e0607668d3924d5cba6acd89b2cdc25f with delete_type TABLET_DATA_TOMBSTONED (TS d4181d4d907f477f9feddc44ba3b9207 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33214
I20260806 20:15:10.262526 30700 tablet_service.cc:1514] Processing DeleteTablet for tablet 01fd0995e74a449090ea2b00062f625e with delete_type TABLET_DATA_TOMBSTONED (TS 42cabca48db7403b8523a3382a046043 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20260806 20:15:10.263983 30713 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:10.266641 31591 tablet_replica.cc:330] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: stopping tablet replica
I20260806 20:15:10.267441 30789 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:10.268041 31592 tablet_replica.cc:330] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043: stopping tablet replica
I20260806 20:15:10.269073 31592 raft_consensus.cc:2229] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:10.269743 31592 raft_consensus.cc:2258] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:10.275408 31567 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:10.277210 31567 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:10.284778 31592 ts_tablet_manager.cc:1905] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:10.297762 31422 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:10.302891 31582 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:10.308609 31591 raft_consensus.cc:2229] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:10.309485 31591 raft_consensus.cc:2258] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:10.313366 31592 ts_tablet_manager.cc:1918] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:10.314191 31592 log.cc:1196] T 01fd0995e74a449090ea2b00062f625e P 42cabca48db7403b8523a3382a046043: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/wals/01fd0995e74a449090ea2b00062f625e
I20260806 20:15:10.313657 31582 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.316498 30589 catalog_manager.cc:4841] TS 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): tablet 01fd0995e74a449090ea2b00062f625e (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:10.316097 30713 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.326958 31591 ts_tablet_manager.cc:1905] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:10.337396 30789 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.344761 30603 catalog_manager.cc:5495] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.354545 31422 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.365193 31591 ts_tablet_manager.cc:1918] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:10.365842 31591 log.cc:1196] T e0607668d3924d5cba6acd89b2cdc25f P d4181d4d907f477f9feddc44ba3b9207: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/wals/e0607668d3924d5cba6acd89b2cdc25f
I20260806 20:15:10.367794 30590 catalog_manager.cc:4841] TS d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): tablet e0607668d3924d5cba6acd89b2cdc25f (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:10.392724 30866 consensus_queue.cc:237] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:10.398592 30789 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:10.399730 31525 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:10.399900 31422 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:10.401046 31582 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:10.405730 31582 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.406960 30787 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.412894 31422 raft_consensus.cc:2932] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:10.417662 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7c541d91c0c4487c9a76c6fc6f81a95a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:10.423243 30600 catalog_manager.cc:5495] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:10.433846 30700 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c541d91c0c4487c9a76c6fc6f81a95a with delete_type TABLET_DATA_TOMBSTONED (TS 42cabca48db7403b8523a3382a046043 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20260806 20:15:10.434689 31592 tablet_replica.cc:330] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043: stopping tablet replica
I20260806 20:15:10.435509 31592 raft_consensus.cc:2229] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:10.435981 31592 raft_consensus.cc:2258] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:10.438516 31592 ts_tablet_manager.cc:1905] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:10.451125 31592 ts_tablet_manager.cc:1918] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:10.451599 31592 log.cc:1196] T 7c541d91c0c4487c9a76c6fc6f81a95a P 42cabca48db7403b8523a3382a046043: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/wals/7c541d91c0c4487c9a76c6fc6f81a95a
I20260806 20:15:10.453478 30589 catalog_manager.cc:4841] TS 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): tablet 7c541d91c0c4487c9a76c6fc6f81a95a (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:11.482333 30790 consensus_queue.cc:237] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:11.487851 30866 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:11.487851 30711 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:15:11.488412 30754 consensus_peers.cc:487] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e39c6ca59dfb42bda66c72b0bcf52faa. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:11.489713 31596 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:11.490449 31564 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:11.496927 31501 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.498346 30866 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.498600 30711 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.514021 30866 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:11.517560 30603 catalog_manager.cc:5495] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:11.525032 30790 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:11.525031 30711 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:15:11.525847 30832 consensus_peers.cc:487] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f53758bb873349479f86e0abf34c738b. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:11.528416 31567 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:11.530418 31567 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:11.539996 31567 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.541831 30713 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.549669 30790 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.565917 30600 catalog_manager.cc:5495] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.567076 30866 consensus_queue.cc:237] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:11.582813 30832 consensus_peers.cc:487] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78bef7be8f95499c84c37311213c95b6. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:11.583925 30790 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:11.585016 30713 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:11.585489 31567 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:11.586395 31582 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:11.597710 31605 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.606142 30790 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.622117 30711 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.625574 30713 consensus_queue.cc:237] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:11.626111 30601 catalog_manager.cc:5495] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 20:15:11.638778 30673 consensus_peers.cc:487] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cb9334179bc40b1b3868fc4773af362. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:11.639395 30790 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:11.639861 30867 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:11.640929 31585 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:11.641834 31597 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:11.650923 31566 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.653245 30867 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.665911 30787 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.671468 30601 catalog_manager.cc:5495] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:11.951515 31618 ts_tablet_manager.cc:927] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:11.956385 31618 tablet_copy_client.cc:316] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.195:41735
I20260806 20:15:11.957938 30882 tablet_copy_service.cc:140] P 58d59beb597f4ed99d84d102ddb41ff5: Received BeginTabletCopySession request for tablet f53758bb873349479f86e0abf34c738b from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:45134)
I20260806 20:15:11.958441 30882 tablet_copy_service.cc:161] P 58d59beb597f4ed99d84d102ddb41ff5: Beginning new tablet copy session on tablet f53758bb873349479f86e0abf34c738b from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:45134: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-f53758bb873349479f86e0abf34c738b
I20260806 20:15:11.976450 30882 tablet_copy_source_session.cc:205] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:11.980078 31618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53758bb873349479f86e0abf34c738b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:11.996531 31618 tablet_copy_client.cc:791] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:11.997213 31618 tablet_copy_client.cc:655] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:12.000895 31618 tablet_copy_client.cc:531] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:12.002461 31620 ts_tablet_manager.cc:927] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:12.005426 31620 tablet_copy_client.cc:316] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.195:41735
I20260806 20:15:12.006986 30882 tablet_copy_service.cc:140] P 58d59beb597f4ed99d84d102ddb41ff5: Received BeginTabletCopySession request for tablet 78bef7be8f95499c84c37311213c95b6 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:45134)
I20260806 20:15:12.007455 30882 tablet_copy_service.cc:161] P 58d59beb597f4ed99d84d102ddb41ff5: Beginning new tablet copy session on tablet 78bef7be8f95499c84c37311213c95b6 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:45134: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-78bef7be8f95499c84c37311213c95b6
I20260806 20:15:12.015501 30882 tablet_copy_source_session.cc:205] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:12.021945 31618 tablet_bootstrap.cc:492] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:12.028512 31620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78bef7be8f95499c84c37311213c95b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:12.043092 31618 tablet_bootstrap.cc:492] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:12.044402 31620 tablet_copy_client.cc:791] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:12.044948 31620 tablet_copy_client.cc:655] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:12.045760 31618 tablet_bootstrap.cc:492] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:12.046597 31618 ts_tablet_manager.cc:1397] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.012s
I20260806 20:15:12.049821 31620 tablet_copy_client.cc:531] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:12.049698 31618 raft_consensus.cc:348] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:12.050417 31618 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:12.050938 31618 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:12.058584 31620 tablet_bootstrap.cc:492] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:12.070320 31618 ts_tablet_manager.cc:1428] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.023s user 0.013s sys 0.010s
I20260806 20:15:12.075567 30882 tablet_copy_service.cc:342] P 58d59beb597f4ed99d84d102ddb41ff5: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-f53758bb873349479f86e0abf34c738b received from {username='slave'} at 127.0.0.1:45134
I20260806 20:15:12.076051 30882 tablet_copy_service.cc:433] P 58d59beb597f4ed99d84d102ddb41ff5: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-f53758bb873349479f86e0abf34c738b on tablet f53758bb873349479f86e0abf34c738b with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:12.080886 31618 ts_tablet_manager.cc:927] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:12.086539 31618 tablet_copy_client.cc:316] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.194:42029
I20260806 20:15:12.088199 30804 tablet_copy_service.cc:140] P d4181d4d907f477f9feddc44ba3b9207: Received BeginTabletCopySession request for tablet e39c6ca59dfb42bda66c72b0bcf52faa from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:53732)
I20260806 20:15:12.088799 30804 tablet_copy_service.cc:161] P d4181d4d907f477f9feddc44ba3b9207: Beginning new tablet copy session on tablet e39c6ca59dfb42bda66c72b0bcf52faa from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:53732: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-e39c6ca59dfb42bda66c72b0bcf52faa
I20260806 20:15:12.094708 31620 tablet_bootstrap.cc:492] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:12.095798 31620 tablet_bootstrap.cc:492] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:12.096480 31620 ts_tablet_manager.cc:1397] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.038s user 0.023s sys 0.004s
I20260806 20:15:12.097095 30804 tablet_copy_source_session.cc:205] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:12.099666 31620 raft_consensus.cc:348] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:12.100396 31618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39c6ca59dfb42bda66c72b0bcf52faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:12.100607 31620 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:12.101259 31620 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:12.167207 31620 ts_tablet_manager.cc:1428] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.070s user 0.033s sys 0.023s
I20260806 20:15:12.184489 30882 tablet_copy_service.cc:342] P 58d59beb597f4ed99d84d102ddb41ff5: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-78bef7be8f95499c84c37311213c95b6 received from {username='slave'} at 127.0.0.1:45134
I20260806 20:15:12.184947 30882 tablet_copy_service.cc:433] P 58d59beb597f4ed99d84d102ddb41ff5: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-78bef7be8f95499c84c37311213c95b6 on tablet 78bef7be8f95499c84c37311213c95b6 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:12.210392 31618 tablet_copy_client.cc:791] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:12.211364 31618 tablet_copy_client.cc:655] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:12.225898 31618 tablet_copy_client.cc:531] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:12.233119 31618 tablet_bootstrap.cc:492] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:12.253620 31620 ts_tablet_manager.cc:927] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:12.255273 31618 tablet_bootstrap.cc:492] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:12.256219 31618 tablet_bootstrap.cc:492] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:12.256922 31618 ts_tablet_manager.cc:1397] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.013s
I20260806 20:15:12.260939 31618 raft_consensus.cc:348] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:12.261955 31618 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:12.261727 31620 tablet_copy_client.cc:316] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:12.262629 31618 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:12.264258 30724 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet 7cb9334179bc40b1b3868fc4773af362 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:12.264761 30724 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet 7cb9334179bc40b1b3868fc4773af362 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-7cb9334179bc40b1b3868fc4773af362
I20260806 20:15:12.271054 31618 ts_tablet_manager.cc:1428] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.014s user 0.002s sys 0.003s
I20260806 20:15:12.272269 30724 tablet_copy_source_session.cc:205] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:12.273195 30804 tablet_copy_service.cc:342] P d4181d4d907f477f9feddc44ba3b9207: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-e39c6ca59dfb42bda66c72b0bcf52faa received from {username='slave'} at 127.0.0.1:53732
I20260806 20:15:12.273569 30804 tablet_copy_service.cc:433] P d4181d4d907f477f9feddc44ba3b9207: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-e39c6ca59dfb42bda66c72b0bcf52faa on tablet e39c6ca59dfb42bda66c72b0bcf52faa with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:12.276739 31620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb9334179bc40b1b3868fc4773af362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:12.291635 31620 tablet_copy_client.cc:791] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:12.292264 31620 tablet_copy_client.cc:655] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:12.296695 31620 tablet_copy_client.cc:531] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:12.304612 31620 tablet_bootstrap.cc:492] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:12.322662 31620 tablet_bootstrap.cc:492] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:12.323482 31620 tablet_bootstrap.cc:492] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:12.324128 31620 ts_tablet_manager.cc:1397] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20260806 20:15:12.326516 31620 raft_consensus.cc:348] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:12.327148 31620 raft_consensus.cc:729] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:12.327672 31620 consensus_queue.cc:260] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:12.330005 31620 ts_tablet_manager.cc:1428] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 20:15:12.331568 30724 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-7cb9334179bc40b1b3868fc4773af362 received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:12.331955 30724 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-7cb9334179bc40b1b3868fc4773af362 on tablet 7cb9334179bc40b1b3868fc4773af362 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:12.482218 31421 raft_consensus.cc:1203] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:12.487088 31422 raft_consensus.cc:1203] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:12.501911 31422 raft_consensus.cc:1203] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:12.859592 31422 raft_consensus.cc:1203] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:12.944317 31564 raft_consensus.cc:1050] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:12.947257 31564 consensus_queue.cc:237] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:12.968513 31422 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:12.970697 31629 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:12.975479 30711 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:12.975874 30866 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:12.977185 31564 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:12.978868 31629 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:12.999001 31627 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.009188 30711 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.015820 31422 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.027269 31614 raft_consensus.cc:1050] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:13.029042 30866 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.030617 31614 consensus_queue.cc:237] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:13.036790 30600 catalog_manager.cc:5495] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 20:15:13.099650 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 52977.3us
I20260806 20:15:13.156291 31421 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:13.156375 30789 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:13.160593 31631 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:13.162712 31630 raft_consensus.cc:1050] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:13.165540 31630 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:13.170408 30711 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:13.171681 31648 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:13.173229 30788 consensus_queue.cc:237] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:13.174829 31647 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:13.179457 31422 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:13.179840 30711 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:13.181306 31627 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:13.187883 31627 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260806 20:15:13.191669 31501 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.192976 30711 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.212342 31422 raft_consensus.cc:2932] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.225587 30600 catalog_manager.cc:5495] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.238736 30792 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:13.239661 31421 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:13.240482 31646 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:13.241951 31582 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:13.241505 31649 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.243902 30792 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.246867 30711 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:13.248299 31648 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:13.255004 30855 tablet_service.cc:1514] Processing DeleteTablet for tablet e39c6ca59dfb42bda66c72b0bcf52faa with delete_type TABLET_DATA_TOMBSTONED (TS 58d59beb597f4ed99d84d102ddb41ff5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60882
I20260806 20:15:13.261096 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e39c6ca59dfb42bda66c72b0bcf52faa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:13.271662 31630 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.272607 30706 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.290225 30603 catalog_manager.cc:5495] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:13.299093 30713 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.301432 31422 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.315495 31657 tablet_replica.cc:330] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5: stopping tablet replica
I20260806 20:15:13.322472 31421 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.332517 31657 raft_consensus.cc:2229] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:13.333559 31657 raft_consensus.cc:2258] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:13.340982 31657 ts_tablet_manager.cc:1905] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:13.346024 30789 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.361651 31657 ts_tablet_manager.cc:1918] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:13.361120 30600 catalog_manager.cc:5495] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.362493 31657 log.cc:1196] T e39c6ca59dfb42bda66c72b0bcf52faa P 58d59beb597f4ed99d84d102ddb41ff5: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/wals/e39c6ca59dfb42bda66c72b0bcf52faa
I20260806 20:15:13.364208 30590 catalog_manager.cc:4841] TS 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): tablet e39c6ca59dfb42bda66c72b0bcf52faa (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:13.346886 30865 consensus_queue.cc:237] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:13.369184 31566 raft_consensus.cc:1050] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:13.375187 31421 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:13.371766 31566 consensus_queue.cc:237] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:13.377851 31630 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:13.393936 30789 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:13.393008 31646 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.396675 31421 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.417107 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 78bef7be8f95499c84c37311213c95b6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:13.419786 30865 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:13.423539 30600 catalog_manager.cc:5495] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.424269 31647 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:13.435336 30700 tablet_service.cc:1514] Processing DeleteTablet for tablet 78bef7be8f95499c84c37311213c95b6 with delete_type TABLET_DATA_TOMBSTONED (TS 42cabca48db7403b8523a3382a046043 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20260806 20:15:13.428905 30789 raft_consensus.cc:2932] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.453744 31421 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:13.458866 31647 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:13.468014 31649 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.471297 31660 tablet_replica.cc:330] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043: stopping tablet replica
I20260806 20:15:13.471109 31422 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.474681 31660 raft_consensus.cc:2229] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:13.475301 31660 raft_consensus.cc:2258] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:13.473217 30706 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:13.473498 31423 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:13.482115 31566 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:13.483525 30790 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:13.484822 31566 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:13.485405 30867 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:13.482159 31630 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:13.486934 31566 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:13.494238 30600 catalog_manager.cc:5495] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.502848 31660 ts_tablet_manager.cc:1905] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:13.501559 30706 raft_consensus.cc:2932] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.504091 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f53758bb873349479f86e0abf34c738b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:13.518411 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet f53758bb873349479f86e0abf34c738b with delete_type TABLET_DATA_TOMBSTONED (TS d4181d4d907f477f9feddc44ba3b9207 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33214
I20260806 20:15:13.527454 31661 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.529035 30867 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.539772 30790 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.540858 31660 ts_tablet_manager.cc:1918] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:13.541386 31660 log.cc:1196] T 78bef7be8f95499c84c37311213c95b6 P 42cabca48db7403b8523a3382a046043: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/wals/78bef7be8f95499c84c37311213c95b6
I20260806 20:15:13.543534 30589 catalog_manager.cc:4841] TS 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): tablet 78bef7be8f95499c84c37311213c95b6 (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:13.551550 31423 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.556298 30602 catalog_manager.cc:5495] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:13.620740 31666 tablet_replica.cc:330] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207: stopping tablet replica
I20260806 20:15:13.627437 31666 raft_consensus.cc:2229] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:13.635313 31666 raft_consensus.cc:2258] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:13.639424 31666 ts_tablet_manager.cc:1905] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:13.651762 30710 consensus_queue.cc:237] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:13.665264 31423 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:13.666960 31661 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:13.678638 31636 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.687177 31423 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.695267 30790 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:13.703737 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7cb9334179bc40b1b3868fc4773af362 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:13.713320 30602 catalog_manager.cc:5495] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.713999 31661 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:13.716791 31666 ts_tablet_manager.cc:1918] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:13.720053 31666 log.cc:1196] T f53758bb873349479f86e0abf34c738b P d4181d4d907f477f9feddc44ba3b9207: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/wals/f53758bb873349479f86e0abf34c738b
I20260806 20:15:13.724502 30590 catalog_manager.cc:4841] TS d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): tablet f53758bb873349479f86e0abf34c738b (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:13.726469 30855 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cb9334179bc40b1b3868fc4773af362 with delete_type TABLET_DATA_TOMBSTONED (TS 58d59beb597f4ed99d84d102ddb41ff5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60882
I20260806 20:15:13.727485 31657 tablet_replica.cc:330] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5: stopping tablet replica
I20260806 20:15:13.728564 31657 raft_consensus.cc:2229] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:13.729375 31657 raft_consensus.cc:2258] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:13.728606 30790 raft_consensus.cc:2932] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:13.734329 31657 ts_tablet_manager.cc:1905] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:13.761215 31657 ts_tablet_manager.cc:1918] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:13.761704 31657 log.cc:1196] T 7cb9334179bc40b1b3868fc4773af362 P 58d59beb597f4ed99d84d102ddb41ff5: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/wals/7cb9334179bc40b1b3868fc4773af362
I20260806 20:15:13.763459 30590 catalog_manager.cc:4841] TS 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): tablet 7cb9334179bc40b1b3868fc4773af362 (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:14.787552 30706 consensus_queue.cc:237] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:14.794216 30867 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:15:14.795648 30673 consensus_peers.cc:487] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 942abb68076e47a0a24672bb92649892. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:14.797653 30790 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:14.799787 31597 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:14.801436 31566 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:14.811019 31597 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.814056 30866 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.817273 30790 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.829115 30789 consensus_queue.cc:237] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:14.844357 30706 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:14.844383 30600 catalog_manager.cc:5495] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:14.846920 31501 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:15:14.850190 30754 consensus_peers.cc:487] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5e69bf76cc845c8822b4cfab14eac47. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:14.854491 30865 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:14.856178 31501 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:14.876580 31673 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.878939 30713 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.880749 30866 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.900744 30706 consensus_queue.cc:237] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:14.911099 30673 consensus_peers.cc:487] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67c52f63d3e8412bbd0e51fe5b81568b. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:14.912528 30865 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:14.914120 31597 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:14.914400 30790 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:14.916512 31597 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:14.927698 30602 catalog_manager.cc:5495] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:14.933422 31566 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.935943 30790 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.938081 30866 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.947719 30706 consensus_queue.cc:237] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:14.957926 30601 catalog_manager.cc:5495] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
W20260806 20:15:14.964748 30673 consensus_peers.cc:487] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0c1793b412c4608b1e004525a6dd887. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:14.964694 30866 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:14.967509 31566 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:14.967928 30789 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:14.971637 31566 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:14.979132 31636 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.981240 30866 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:15.002074 30601 catalog_manager.cc:5495] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:14.997788 30789 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:15.285707 31690 ts_tablet_manager.cc:927] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:15.287710 31690 tablet_copy_client.cc:316] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:15.289484 30724 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet 942abb68076e47a0a24672bb92649892 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:15.290059 30724 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet 942abb68076e47a0a24672bb92649892 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-942abb68076e47a0a24672bb92649892
I20260806 20:15:15.301316 30724 tablet_copy_source_session.cc:205] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:15.304966 31690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 942abb68076e47a0a24672bb92649892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:15.313603 31692 ts_tablet_manager.cc:927] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:15.319680 31690 tablet_copy_client.cc:791] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:15.320348 31690 tablet_copy_client.cc:655] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:15.321521 31692 tablet_copy_client.cc:316] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.194:42029
I20260806 20:15:15.323553 30804 tablet_copy_service.cc:140] P d4181d4d907f477f9feddc44ba3b9207: Received BeginTabletCopySession request for tablet a5e69bf76cc845c8822b4cfab14eac47 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:53732)
I20260806 20:15:15.324093 30804 tablet_copy_service.cc:161] P d4181d4d907f477f9feddc44ba3b9207: Beginning new tablet copy session on tablet a5e69bf76cc845c8822b4cfab14eac47 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:53732: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-a5e69bf76cc845c8822b4cfab14eac47
I20260806 20:15:15.325161 31690 tablet_copy_client.cc:531] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:15.336896 31690 tablet_bootstrap.cc:492] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:15.339542 30804 tablet_copy_source_session.cc:205] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:15.346844 31692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e69bf76cc845c8822b4cfab14eac47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:15.367218 31692 tablet_copy_client.cc:791] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:15.367784 31692 tablet_copy_client.cc:655] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:15.371553 31692 tablet_copy_client.cc:531] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:15.373255 31690 tablet_bootstrap.cc:492] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:15.374174 31690 tablet_bootstrap.cc:492] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:15.374850 31690 ts_tablet_manager.cc:1397] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.007s
I20260806 20:15:15.377811 31690 raft_consensus.cc:348] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:15.378654 31690 raft_consensus.cc:729] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:15.379714 31690 consensus_queue.cc:260] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:15.383102 31690 ts_tablet_manager.cc:1428] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.008s user 0.007s sys 0.002s
I20260806 20:15:15.385830 30724 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-942abb68076e47a0a24672bb92649892 received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:15.386245 30724 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-942abb68076e47a0a24672bb92649892 on tablet 942abb68076e47a0a24672bb92649892 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:15.389259 31690 ts_tablet_manager.cc:927] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:15.395901 31692 tablet_bootstrap.cc:492] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:15.399765 31690 tablet_copy_client.cc:316] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:15.401898 30724 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet e0c1793b412c4608b1e004525a6dd887 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:15.402423 30724 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet e0c1793b412c4608b1e004525a6dd887 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-e0c1793b412c4608b1e004525a6dd887
I20260806 20:15:15.411579 30724 tablet_copy_source_session.cc:205] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:15.415077 31690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c1793b412c4608b1e004525a6dd887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:15.423826 31692 tablet_bootstrap.cc:492] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:15.424774 31692 tablet_bootstrap.cc:492] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:15.425480 31692 ts_tablet_manager.cc:1397] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.004s
I20260806 20:15:15.428282 31692 raft_consensus.cc:348] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:15.428937 31692 raft_consensus.cc:729] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:15.429529 31692 consensus_queue.cc:260] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:15.431469 31692 ts_tablet_manager.cc:1428] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:15:15.434082 31696 ts_tablet_manager.cc:927] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:15.434341 31690 tablet_copy_client.cc:791] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:15.434999 31690 tablet_copy_client.cc:655] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:15.436120 30804 tablet_copy_service.cc:342] P d4181d4d907f477f9feddc44ba3b9207: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-a5e69bf76cc845c8822b4cfab14eac47 received from {username='slave'} at 127.0.0.1:53732
I20260806 20:15:15.436553 30804 tablet_copy_service.cc:433] P d4181d4d907f477f9feddc44ba3b9207: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-a5e69bf76cc845c8822b4cfab14eac47 on tablet a5e69bf76cc845c8822b4cfab14eac47 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:15.436648 31696 tablet_copy_client.cc:316] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:15.439141 30722 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet 67c52f63d3e8412bbd0e51fe5b81568b from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:15.439260 31690 tablet_copy_client.cc:531] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:15.439604 30722 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet 67c52f63d3e8412bbd0e51fe5b81568b from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-67c52f63d3e8412bbd0e51fe5b81568b
I20260806 20:15:15.447063 30722 tablet_copy_source_session.cc:205] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:15.448328 31690 tablet_bootstrap.cc:492] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:15.450268 31696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c52f63d3e8412bbd0e51fe5b81568b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:15.465884 31696 tablet_copy_client.cc:791] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:15.466482 31696 tablet_copy_client.cc:655] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:15.477304 31696 tablet_copy_client.cc:531] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:15.480149 31690 tablet_bootstrap.cc:492] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:15.480950 31690 tablet_bootstrap.cc:492] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:15.481578 31690 ts_tablet_manager.cc:1397] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.004s
I20260806 20:15:15.484045 31690 raft_consensus.cc:348] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:15.484762 31690 raft_consensus.cc:729] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:15.485472 31696 tablet_bootstrap.cc:492] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:15.485301 31690 consensus_queue.cc:260] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:15.487555 31690 ts_tablet_manager.cc:1428] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:15:15.490845 30722 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-e0c1793b412c4608b1e004525a6dd887 received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:15.491227 30722 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-e0c1793b412c4608b1e004525a6dd887 on tablet e0c1793b412c4608b1e004525a6dd887 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:15.517426 31696 tablet_bootstrap.cc:492] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:15.518285 31696 tablet_bootstrap.cc:492] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:15.518990 31696 ts_tablet_manager.cc:1397] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.015s
I20260806 20:15:15.521533 31696 raft_consensus.cc:348] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:15.522333 31696 raft_consensus.cc:729] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:15.522975 31696 consensus_queue.cc:260] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:15.525359 31696 ts_tablet_manager.cc:1428] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:15:15.527146 30722 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-67c52f63d3e8412bbd0e51fe5b81568b received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:15.527539 30722 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-67c52f63d3e8412bbd0e51fe5b81568b on tablet 67c52f63d3e8412bbd0e51fe5b81568b with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:15.798866 31422 raft_consensus.cc:1203] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:15.899389 31422 raft_consensus.cc:1203] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:15.926564 31422 raft_consensus.cc:1203] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260806 20:15:15.979324 31422 raft_consensus.cc:1203] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:16.355598 31566 raft_consensus.cc:1050] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:16.356225 31704 raft_consensus.cc:1050] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:16.358052 31566 consensus_queue.cc:237] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:16.358716 31704 consensus_queue.cc:237] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:16.374189 30865 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:16.374927 31423 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:16.375918 30792 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:16.376955 31423 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:16.378154 31702 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:16.379755 31708 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:16.380087 31702 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:16.384159 31636 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:16.384984 30788 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:16.386459 30865 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:16.386480 31702 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260806 20:15:16.396672 31636 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 20:15:16.412045 31566 raft_consensus.cc:1050] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:16.414458 31566 consensus_queue.cc:237] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:16.441520 31704 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.441970 31422 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.445056 31423 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:16.444734 30792 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.445336 31636 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.451664 31702 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:16.442294 30790 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.461700 31676 raft_consensus.cc:1050] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:16.465843 30601 catalog_manager.cc:5495] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.464553 31676 consensus_queue.cc:237] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:16.452793 31423 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:15:16.537439 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 53967.9us
I20260806 20:15:16.575165 30865 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 20:15:16.578744 31422 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEARNER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 20:15:16.592024 31676 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:16.593032 30602 catalog_manager.cc:5495] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.604246 30792 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:16.605290 31676 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:16.614784 30710 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 20:15:16.644731 31716 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:16.648371 31722 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260806 20:15:16.677489 30866 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.679636 30867 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.700063 30869 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260806 20:15:16.691445 31723 log.cc:924] Time spent T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
W20260806 20:15:16.717120 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 82949.3us
I20260806 20:15:16.733948 31636 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 20:15:16.772280 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 46290.3us
I20260806 20:15:16.778837 31716 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.779121 31722 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.782332 30790 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.796855 31422 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.798349 31423 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:15:16.805263 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 146321us
I20260806 20:15:16.793748 30710 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.819196 30865 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.843740 30865 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.851958 30706 consensus_queue.cc:237] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:16.858104 30601 catalog_manager.cc:5495] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:16.860210 30601 catalog_manager.cc:5495] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.861199 31422 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:16.864166 31738 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:16.872471 30865 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:16.870536 31738 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.875528 31636 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:16.880578 30865 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.884903 30787 consensus_queue.cc:237] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:16.888012 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 67c52f63d3e8412bbd0e51fe5b81568b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:16.881680 31422 raft_consensus.cc:2932] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.890379 31423 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 20:15:16.891609 31722 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:16.898490 31722 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.903546 30713 consensus_queue.cc:237] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:16.903750 30710 consensus_queue.cc:237] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:16.904544 30865 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 20:15:16.907557 31423 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.908702 31719 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:16.909750 31419 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:16.911358 31704 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:16.912595 31419 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:16.914217 31704 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:16.915802 30787 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:16.917595 31704 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:16.920365 30871 raft_consensus.cc:2932] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.926129 31419 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.927946 30602 catalog_manager.cc:5495] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:16.931892 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a5e69bf76cc845c8822b4cfab14eac47 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:16.921518 30787 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:16.924378 31708 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.936739 31702 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.947500 30600 catalog_manager.cc:5495] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:16.948681 31566 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:16.953717 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 942abb68076e47a0a24672bb92649892 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:16.960017 31421 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.967178 30787 raft_consensus.cc:2932] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:16.972389 30700 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e69bf76cc845c8822b4cfab14eac47 with delete_type TABLET_DATA_TOMBSTONED (TS 42cabca48db7403b8523a3382a046043 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20260806 20:15:16.974426 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e0c1793b412c4608b1e004525a6dd887 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:16.988174 31742 tablet_replica.cc:330] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043: stopping tablet replica
I20260806 20:15:16.993585 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c52f63d3e8412bbd0e51fe5b81568b with delete_type TABLET_DATA_TOMBSTONED (TS d4181d4d907f477f9feddc44ba3b9207 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33214
I20260806 20:15:16.996449 31743 tablet_replica.cc:330] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207: stopping tablet replica
I20260806 20:15:16.997901 31743 raft_consensus.cc:2229] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:16.999380 31743 raft_consensus.cc:2258] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:17.004136 31743 ts_tablet_manager.cc:1905] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:17.015939 30599 catalog_manager.cc:5495] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:16.987421 30790 raft_consensus.cc:2932] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:17.072451 31742 raft_consensus.cc:2229] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:17.073774 31742 raft_consensus.cc:2258] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:17.079273 31742 ts_tablet_manager.cc:1905] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:17.072808 30599 catalog_manager.cc:5495] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:17.095466 31743 ts_tablet_manager.cc:1918] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:17.096151 31743 log.cc:1196] T 67c52f63d3e8412bbd0e51fe5b81568b P d4181d4d907f477f9feddc44ba3b9207: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/wals/67c52f63d3e8412bbd0e51fe5b81568b
I20260806 20:15:17.098152 30590 catalog_manager.cc:4841] TS d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): tablet 67c52f63d3e8412bbd0e51fe5b81568b (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:17.098805 31742 ts_tablet_manager.cc:1918] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260806 20:15:17.099329 31742 log.cc:1196] T a5e69bf76cc845c8822b4cfab14eac47 P 42cabca48db7403b8523a3382a046043: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/wals/a5e69bf76cc845c8822b4cfab14eac47
I20260806 20:15:17.100975 30589 catalog_manager.cc:4841] TS 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): tablet a5e69bf76cc845c8822b4cfab14eac47 (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:17.110950 30855 tablet_service.cc:1514] Processing DeleteTablet for tablet 942abb68076e47a0a24672bb92649892 with delete_type TABLET_DATA_TOMBSTONED (TS 58d59beb597f4ed99d84d102ddb41ff5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60882
I20260806 20:15:17.111244 30856 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c1793b412c4608b1e004525a6dd887 with delete_type TABLET_DATA_TOMBSTONED (TS 58d59beb597f4ed99d84d102ddb41ff5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60882
I20260806 20:15:17.112792 31744 tablet_replica.cc:330] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5: stopping tablet replica
I20260806 20:15:17.113895 31744 raft_consensus.cc:2229] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:17.114586 31744 raft_consensus.cc:2258] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:17.117297 31744 ts_tablet_manager.cc:1905] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:17.130102 31744 ts_tablet_manager.cc:1918] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:17.130491 31744 log.cc:1196] T 942abb68076e47a0a24672bb92649892 P 58d59beb597f4ed99d84d102ddb41ff5: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/wals/942abb68076e47a0a24672bb92649892
I20260806 20:15:17.132282 31744 tablet_replica.cc:330] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5: stopping tablet replica
I20260806 20:15:17.132218 30590 catalog_manager.cc:4841] TS 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): tablet 942abb68076e47a0a24672bb92649892 (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:17.133240 31744 raft_consensus.cc:2229] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:17.134493 31744 raft_consensus.cc:2258] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:17.142108 31744 ts_tablet_manager.cc:1905] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:17.154590 31744 ts_tablet_manager.cc:1918] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:17.154942 31744 log.cc:1196] T e0c1793b412c4608b1e004525a6dd887 P 58d59beb597f4ed99d84d102ddb41ff5: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/wals/e0c1793b412c4608b1e004525a6dd887
I20260806 20:15:17.156034 30590 catalog_manager.cc:4841] TS 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): tablet e0c1793b412c4608b1e004525a6dd887 (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:18.053560 30865 consensus_queue.cc:237] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.059830 30787 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:18.060755 30710 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:18.061190 31733 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:18.062384 31729 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:15:18.061419 30832 consensus_peers.cc:487] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1066045a99d4a1da0fff49e2b8bbe30. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:18.071630 31729 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.073311 30787 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.075647 30710 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.088245 30706 consensus_queue.cc:237] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.095534 30603 catalog_manager.cc:5495] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 20:15:18.124668 30673 consensus_peers.cc:487] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47c09420eb1e4f3692c165f76cd1a7d4. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:18.125025 30865 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:18.126595 31636 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:18.133126 30787 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:18.134701 31636 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:18.136598 31636 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.139765 30865 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.152704 30871 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:18.165843 30832 consensus_peers.cc:487] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90f7b0d94b4e4047a32e7fbdde6c35d5. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:18.166545 30706 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:18.166767 30790 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:18.167909 31729 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:18.168814 31746 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:18.171696 30603 catalog_manager.cc:5495] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:18.182705 30790 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.186802 31733 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.188400 30710 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.206947 30790 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.212738 30603 catalog_manager.cc:5495] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:18.211668 30787 consensus_queue.cc:237] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:18.224175 30754 consensus_peers.cc:487] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48c353d311a64e628bfc1af985e1d8ba. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:18.226155 30865 raft_consensus.cc:1261] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:18.228683 31719 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:18.235564 30710 raft_consensus.cc:1261] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:18.237011 31767 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:18.253160 31767 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.258675 30706 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.256997 30865 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.279841 30599 catalog_manager.cc:5495] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:18.437965 31771 ts_tablet_manager.cc:927] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:18.452944 31771 tablet_copy_client.cc:316] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.195:41735
I20260806 20:15:18.455072 30882 tablet_copy_service.cc:140] P 58d59beb597f4ed99d84d102ddb41ff5: Received BeginTabletCopySession request for tablet a1066045a99d4a1da0fff49e2b8bbe30 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:45134)
I20260806 20:15:18.455623 30882 tablet_copy_service.cc:161] P 58d59beb597f4ed99d84d102ddb41ff5: Beginning new tablet copy session on tablet a1066045a99d4a1da0fff49e2b8bbe30 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:45134: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-a1066045a99d4a1da0fff49e2b8bbe30
I20260806 20:15:18.463138 30882 tablet_copy_source_session.cc:205] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:18.467039 31771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1066045a99d4a1da0fff49e2b8bbe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:18.481304 31771 tablet_copy_client.cc:791] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:18.481885 31771 tablet_copy_client.cc:655] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:18.485102 31771 tablet_copy_client.cc:531] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:18.491861 31771 tablet_bootstrap.cc:492] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:18.508750 31771 tablet_bootstrap.cc:492] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:18.509627 31771 tablet_bootstrap.cc:492] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:18.510278 31771 ts_tablet_manager.cc:1397] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.008s
I20260806 20:15:18.512946 31771 raft_consensus.cc:348] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.513689 31771 raft_consensus.cc:729] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:18.514261 31771 consensus_queue.cc:260] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.517330 31771 ts_tablet_manager.cc:1428] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:15:18.519296 30882 tablet_copy_service.cc:342] P 58d59beb597f4ed99d84d102ddb41ff5: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-a1066045a99d4a1da0fff49e2b8bbe30 received from {username='slave'} at 127.0.0.1:45134
I20260806 20:15:18.519675 30882 tablet_copy_service.cc:433] P 58d59beb597f4ed99d84d102ddb41ff5: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-a1066045a99d4a1da0fff49e2b8bbe30 on tablet a1066045a99d4a1da0fff49e2b8bbe30 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:18.542105 31771 ts_tablet_manager.cc:927] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:18.543730 31771 tablet_copy_client.cc:316] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:18.545379 30722 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet 47c09420eb1e4f3692c165f76cd1a7d4 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:18.545929 30722 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet 47c09420eb1e4f3692c165f76cd1a7d4 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-47c09420eb1e4f3692c165f76cd1a7d4
I20260806 20:15:18.552493 30722 tablet_copy_source_session.cc:205] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:18.557519 31771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c09420eb1e4f3692c165f76cd1a7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:18.573853 31771 tablet_copy_client.cc:791] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:18.574446 31771 tablet_copy_client.cc:655] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:18.577870 31771 tablet_copy_client.cc:531] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:18.586128 31771 tablet_bootstrap.cc:492] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:18.604151 31771 tablet_bootstrap.cc:492] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:18.604950 31771 tablet_bootstrap.cc:492] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:18.605623 31771 ts_tablet_manager.cc:1397] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20260806 20:15:18.608037 31771 raft_consensus.cc:348] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.608927 31771 raft_consensus.cc:729] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:18.609581 31771 consensus_queue.cc:260] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.612433 31771 ts_tablet_manager.cc:1428] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:15:18.614202 30722 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-47c09420eb1e4f3692c165f76cd1a7d4 received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:18.614563 30722 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-47c09420eb1e4f3692c165f76cd1a7d4 on tablet 47c09420eb1e4f3692c165f76cd1a7d4 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:18.617833 31771 ts_tablet_manager.cc:927] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:18.628051 31771 tablet_copy_client.cc:316] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.194:42029
I20260806 20:15:18.629611 30804 tablet_copy_service.cc:140] P d4181d4d907f477f9feddc44ba3b9207: Received BeginTabletCopySession request for tablet 48c353d311a64e628bfc1af985e1d8ba from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:53732)
I20260806 20:15:18.630129 30804 tablet_copy_service.cc:161] P d4181d4d907f477f9feddc44ba3b9207: Beginning new tablet copy session on tablet 48c353d311a64e628bfc1af985e1d8ba from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:53732: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-48c353d311a64e628bfc1af985e1d8ba
I20260806 20:15:18.641551 30804 tablet_copy_source_session.cc:205] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:18.648571 31771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c353d311a64e628bfc1af985e1d8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:18.656788 31777 ts_tablet_manager.cc:927] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:18.662783 31777 tablet_copy_client.cc:316] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.195:41735
I20260806 20:15:18.664165 31771 tablet_copy_client.cc:791] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:18.664286 30882 tablet_copy_service.cc:140] P 58d59beb597f4ed99d84d102ddb41ff5: Received BeginTabletCopySession request for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:45134)
I20260806 20:15:18.664842 31771 tablet_copy_client.cc:655] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:18.664896 30882 tablet_copy_service.cc:161] P 58d59beb597f4ed99d84d102ddb41ff5: Beginning new tablet copy session on tablet 90f7b0d94b4e4047a32e7fbdde6c35d5 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:45134: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-90f7b0d94b4e4047a32e7fbdde6c35d5
I20260806 20:15:18.669555 31771 tablet_copy_client.cc:531] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:18.674944 30882 tablet_copy_source_session.cc:205] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:18.678936 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:18.679848 31771 tablet_bootstrap.cc:492] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:18.691797 31777 tablet_copy_client.cc:791] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:18.692487 31777 tablet_copy_client.cc:655] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:18.695843 31777 tablet_copy_client.cc:531] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:18.698544 31771 tablet_bootstrap.cc:492] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:18.699486 31771 tablet_bootstrap.cc:492] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:18.700152 31771 ts_tablet_manager.cc:1397] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20260806 20:15:18.704559 31771 raft_consensus.cc:348] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.705286 31777 tablet_bootstrap.cc:492] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:18.705384 31771 raft_consensus.cc:729] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:18.706156 31771 consensus_queue.cc:260] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.818361 31771 ts_tablet_manager.cc:1428] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.118s user 0.007s sys 0.000s
I20260806 20:15:18.870632 31777 tablet_bootstrap.cc:492] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:18.870693 30804 tablet_copy_service.cc:342] P d4181d4d907f477f9feddc44ba3b9207: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-48c353d311a64e628bfc1af985e1d8ba received from {username='slave'} at 127.0.0.1:53732
I20260806 20:15:18.871376 30804 tablet_copy_service.cc:433] P d4181d4d907f477f9feddc44ba3b9207: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-48c353d311a64e628bfc1af985e1d8ba on tablet 48c353d311a64e628bfc1af985e1d8ba with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:18.871655 31777 tablet_bootstrap.cc:492] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:18.873585 31777 ts_tablet_manager.cc:1397] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.169s user 0.041s sys 0.033s
I20260806 20:15:18.878175 31777 raft_consensus.cc:348] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.879089 31777 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:18.879781 31777 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:18.881834 31777 ts_tablet_manager.cc:1428] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:15:18.889060 30882 tablet_copy_service.cc:342] P 58d59beb597f4ed99d84d102ddb41ff5: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-90f7b0d94b4e4047a32e7fbdde6c35d5 received from {username='slave'} at 127.0.0.1:45134
I20260806 20:15:18.889528 30882 tablet_copy_service.cc:433] P 58d59beb597f4ed99d84d102ddb41ff5: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-90f7b0d94b4e4047a32e7fbdde6c35d5 on tablet 90f7b0d94b4e4047a32e7fbdde6c35d5 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:18.902990 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 78902.1us
I20260806 20:15:18.909473 31419 raft_consensus.cc:1203] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:19.150421 31419 raft_consensus.cc:1203] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:19.191236 31419 raft_consensus.cc:1203] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:19.204409 31419 raft_consensus.cc:1203] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:19.371061 31749 raft_consensus.cc:1050] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:19.373440 31749 consensus_queue.cc:237] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:19.380261 30787 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.380643 31419 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.381917 30706 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.381996 31746 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:19.382923 31785 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:19.383838 31729 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:19.404618 31785 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.406538 30787 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.410889 31419 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.422160 30706 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.439111 30599 catalog_manager.cc:5495] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.498075 30866 consensus_queue.cc:237] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:19.504976 31419 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:19.506670 31749 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:19.513919 30787 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:19.516546 31729 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:19.518277 31746 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.521018 30790 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.526520 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a1066045a99d4a1da0fff49e2b8bbe30 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:19.527447 31419 raft_consensus.cc:2932] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.536468 30602 catalog_manager.cc:5495] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:19.554445 30700 tablet_service.cc:1514] Processing DeleteTablet for tablet a1066045a99d4a1da0fff49e2b8bbe30 with delete_type TABLET_DATA_TOMBSTONED (TS 42cabca48db7403b8523a3382a046043 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20260806 20:15:19.556478 31797 tablet_replica.cc:330] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043: stopping tablet replica
I20260806 20:15:19.557585 31797 raft_consensus.cc:2229] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:19.558455 31797 raft_consensus.cc:2258] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:19.567004 31797 ts_tablet_manager.cc:1905] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:19.582647 31797 ts_tablet_manager.cc:1918] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:19.583011 31797 log.cc:1196] T a1066045a99d4a1da0fff49e2b8bbe30 P 42cabca48db7403b8523a3382a046043: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/wals/a1066045a99d4a1da0fff49e2b8bbe30
I20260806 20:15:19.584219 31501 raft_consensus.cc:1050] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:19.584823 30589 catalog_manager.cc:4841] TS 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): tablet a1066045a99d4a1da0fff49e2b8bbe30 (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:19.586483 31501 consensus_queue.cc:237] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:19.592341 31419 raft_consensus.cc:1261] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.593314 30866 raft_consensus.cc:1261] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.594216 30710 raft_consensus.cc:1261] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.594234 31501 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:19.595649 31769 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:19.597404 31719 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:19.609165 31501 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.610503 30710 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.615418 30865 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.623991 31762 raft_consensus.cc:1050] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:19.628165 30601 catalog_manager.cc:5495] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.631165 31762 consensus_queue.cc:237] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:19.639703 31419 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.643157 31423 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.646644 31793 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:19.647960 30866 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.649386 31787 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:19.653405 30788 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.655308 31787 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:19.673370 30790 consensus_queue.cc:237] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:19.684063 31419 raft_consensus.cc:1261] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:19.685520 31501 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:19.688119 30865 raft_consensus.cc:1261] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:19.688005 31793 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.691821 31419 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.696733 31719 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:19.699819 31719 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.707423 30601 catalog_manager.cc:5495] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.713409 30790 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.714311 30866 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.713703 30865 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.735176 31419 raft_consensus.cc:2932] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.753671 31749 raft_consensus.cc:1050] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:19.756201 31749 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:19.758143 30602 catalog_manager.cc:5495] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.766269 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 48c353d311a64e628bfc1af985e1d8ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:19.773316 30710 consensus_queue.cc:237] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:19.782939 30700 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c353d311a64e628bfc1af985e1d8ba with delete_type TABLET_DATA_TOMBSTONED (TS 42cabca48db7403b8523a3382a046043 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20260806 20:15:19.783342 31419 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:19.783882 30866 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:19.784984 31797 tablet_replica.cc:330] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043: stopping tablet replica
I20260806 20:15:19.784950 31793 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:19.786207 31797 raft_consensus.cc:2229] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:19.786922 31797 raft_consensus.cc:2258] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:19.792105 31793 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:19.795487 31793 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.798336 31797 ts_tablet_manager.cc:1905] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:19.798926 31423 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.811995 30866 raft_consensus.cc:2932] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.815439 31421 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.818725 31785 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:19.835765 30788 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.838598 31797 ts_tablet_manager.cc:1918] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:19.838352 31729 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:19.839907 30599 catalog_manager.cc:5495] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.841580 31797 log.cc:1196] T 48c353d311a64e628bfc1af985e1d8ba P 42cabca48db7403b8523a3382a046043: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/wals/48c353d311a64e628bfc1af985e1d8ba
I20260806 20:15:19.846051 30710 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:19.847442 30589 catalog_manager.cc:4841] TS 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): tablet 48c353d311a64e628bfc1af985e1d8ba (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:19.848806 31749 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260806 20:15:19.853428 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 47c09420eb1e4f3692c165f76cd1a7d4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:19.869498 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c09420eb1e4f3692c165f76cd1a7d4 with delete_type TABLET_DATA_TOMBSTONED (TS d4181d4d907f477f9feddc44ba3b9207 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33214
I20260806 20:15:19.872083 31729 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.874230 30713 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.885088 31423 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.896586 31811 tablet_replica.cc:330] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207: stopping tablet replica
I20260806 20:15:19.897673 31811 raft_consensus.cc:2229] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:19.898398 31811 raft_consensus.cc:2258] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:19.901944 31811 ts_tablet_manager.cc:1905] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:19.905098 30599 catalog_manager.cc:5495] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.935469 30787 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.942806 31811 ts_tablet_manager.cc:1918] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:19.943403 31811 log.cc:1196] T 47c09420eb1e4f3692c165f76cd1a7d4 P d4181d4d907f477f9feddc44ba3b9207: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/wals/47c09420eb1e4f3692c165f76cd1a7d4
I20260806 20:15:19.945493 30590 catalog_manager.cc:4841] TS d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): tablet 47c09420eb1e4f3692c165f76cd1a7d4 (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:19.964653 30865 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:19.969436 30713 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:19.969774 31423 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:19.971288 31749 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:19.972992 31749 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:19.980742 31785 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.987354 31421 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.988040 30713 raft_consensus.cc:2932] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:19.998174 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:20.002452 30601 catalog_manager.cc:5495] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:20.018587 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f7b0d94b4e4047a32e7fbdde6c35d5 with delete_type TABLET_DATA_TOMBSTONED (TS d4181d4d907f477f9feddc44ba3b9207 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33214
I20260806 20:15:20.019501 31811 tablet_replica.cc:330] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207: stopping tablet replica
I20260806 20:15:20.020493 31811 raft_consensus.cc:2229] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:20.021131 31811 raft_consensus.cc:2258] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:20.024204 31811 ts_tablet_manager.cc:1905] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:20.041934 31811 ts_tablet_manager.cc:1918] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:20.042258 31811 log.cc:1196] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P d4181d4d907f477f9feddc44ba3b9207: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/wals/90f7b0d94b4e4047a32e7fbdde6c35d5
I20260806 20:15:20.043387 30590 catalog_manager.cc:4841] TS d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): tablet 90f7b0d94b4e4047a32e7fbdde6c35d5 (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:21.077860 30710 consensus_queue.cc:237] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.089681 30866 raft_consensus.cc:1261] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 20:15:21.091310 30673 consensus_peers.cc:487] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28d6e84eb2874b2b9dca3387b3d98eff. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:21.091462 30788 raft_consensus.cc:1261] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:21.092067 31782 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:21.093696 31762 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:21.102402 31816 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.104923 30788 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.117292 30865 consensus_queue.cc:237] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.120209 30599 catalog_manager.cc:5495] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:21.116268 30866 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
W20260806 20:15:21.124747 30832 consensus_peers.cc:487] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88608755c61d42ec8ccc0c3c8e536bd6. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:21.128012 30787 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:21.129558 31814 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:21.139264 30713 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:21.140720 31814 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:21.145143 31785 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.148204 30787 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.167850 30788 consensus_queue.cc:237] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.170135 30713 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.183168 30601 catalog_manager.cc:5495] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 20:15:21.185810 30754 consensus_peers.cc:487] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 412dab7986704f48be75b1fdbdccc9fa. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:21.189694 30865 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:21.191252 30713 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:21.191426 31719 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:21.192692 31806 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:21.212800 31813 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.220772 30866 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.241477 30601 catalog_manager.cc:5495] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:21.244076 30710 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.244804 30713 consensus_queue.cc:237] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.265287 30866 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:21.269873 30788 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:21.271756 31762 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:21.273281 31816 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:15:21.274212 30673 consensus_peers.cc:487] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24eb40e43f13412e8f080b6b6ca292de. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:21.283870 31762 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.285585 30788 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.286123 30866 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.301285 30602 catalog_manager.cc:5495] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:21.521363 31836 ts_tablet_manager.cc:927] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:21.527493 31836 tablet_copy_client.cc:316] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.195:41735
I20260806 20:15:21.531059 30882 tablet_copy_service.cc:140] P 58d59beb597f4ed99d84d102ddb41ff5: Received BeginTabletCopySession request for tablet 88608755c61d42ec8ccc0c3c8e536bd6 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:45134)
I20260806 20:15:21.531615 30882 tablet_copy_service.cc:161] P 58d59beb597f4ed99d84d102ddb41ff5: Beginning new tablet copy session on tablet 88608755c61d42ec8ccc0c3c8e536bd6 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:45134: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-88608755c61d42ec8ccc0c3c8e536bd6
I20260806 20:15:21.539266 30882 tablet_copy_source_session.cc:205] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:21.542248 31836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88608755c61d42ec8ccc0c3c8e536bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:21.555166 31836 tablet_copy_client.cc:791] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:21.555696 31836 tablet_copy_client.cc:655] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:21.560629 31836 tablet_copy_client.cc:531] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:21.568728 31836 tablet_bootstrap.cc:492] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:21.587646 31836 tablet_bootstrap.cc:492] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:21.588567 31836 tablet_bootstrap.cc:492] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:21.589272 31836 ts_tablet_manager.cc:1397] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20260806 20:15:21.589355 31839 ts_tablet_manager.cc:927] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:21.591993 31836 raft_consensus.cc:348] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.592860 31836 raft_consensus.cc:729] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:21.593515 31836 consensus_queue.cc:260] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.603953 31836 ts_tablet_manager.cc:1428] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260806 20:15:21.609002 30882 tablet_copy_service.cc:342] P 58d59beb597f4ed99d84d102ddb41ff5: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-88608755c61d42ec8ccc0c3c8e536bd6 received from {username='slave'} at 127.0.0.1:45134
I20260806 20:15:21.609387 30882 tablet_copy_service.cc:433] P 58d59beb597f4ed99d84d102ddb41ff5: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-88608755c61d42ec8ccc0c3c8e536bd6 on tablet 88608755c61d42ec8ccc0c3c8e536bd6 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:21.625952 31839 tablet_copy_client.cc:316] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:21.627867 30722 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet 28d6e84eb2874b2b9dca3387b3d98eff from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:21.628404 30722 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet 28d6e84eb2874b2b9dca3387b3d98eff from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-28d6e84eb2874b2b9dca3387b3d98eff
I20260806 20:15:21.638075 30722 tablet_copy_source_session.cc:205] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:21.640980 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d6e84eb2874b2b9dca3387b3d98eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:21.656571 31839 tablet_copy_client.cc:791] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:21.657173 31839 tablet_copy_client.cc:655] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:21.660662 31839 tablet_copy_client.cc:531] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:21.668960 31839 tablet_bootstrap.cc:492] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:21.686448 31839 tablet_bootstrap.cc:492] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:21.687224 31839 tablet_bootstrap.cc:492] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:21.687888 31839 ts_tablet_manager.cc:1397] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20260806 20:15:21.690809 31839 raft_consensus.cc:348] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.691570 31839 raft_consensus.cc:729] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:21.692109 31839 consensus_queue.cc:260] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.694756 31839 ts_tablet_manager.cc:1428] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 20:15:21.696260 30722 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-28d6e84eb2874b2b9dca3387b3d98eff received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:21.696641 30722 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-28d6e84eb2874b2b9dca3387b3d98eff on tablet 28d6e84eb2874b2b9dca3387b3d98eff with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:21.710769 31839 ts_tablet_manager.cc:927] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:21.712294 31839 tablet_copy_client.cc:316] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.194:42029
I20260806 20:15:21.713667 30804 tablet_copy_service.cc:140] P d4181d4d907f477f9feddc44ba3b9207: Received BeginTabletCopySession request for tablet 412dab7986704f48be75b1fdbdccc9fa from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:53732)
I20260806 20:15:21.714190 30804 tablet_copy_service.cc:161] P d4181d4d907f477f9feddc44ba3b9207: Beginning new tablet copy session on tablet 412dab7986704f48be75b1fdbdccc9fa from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:53732: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-412dab7986704f48be75b1fdbdccc9fa
I20260806 20:15:21.721189 30804 tablet_copy_source_session.cc:205] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:21.723938 31839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412dab7986704f48be75b1fdbdccc9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:21.737764 31839 tablet_copy_client.cc:791] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:21.738307 31839 tablet_copy_client.cc:655] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:21.742856 31839 tablet_copy_client.cc:531] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:21.750020 31839 tablet_bootstrap.cc:492] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:21.763161 31836 ts_tablet_manager.cc:927] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:21.771076 31839 tablet_bootstrap.cc:492] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:21.772094 31839 tablet_bootstrap.cc:492] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:21.772826 31839 ts_tablet_manager.cc:1397] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.001s
I20260806 20:15:21.775382 31839 raft_consensus.cc:348] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.776292 31839 raft_consensus.cc:729] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:21.776808 31839 consensus_queue.cc:260] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.780920 31839 ts_tablet_manager.cc:1428] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 20:15:21.782447 30804 tablet_copy_service.cc:342] P d4181d4d907f477f9feddc44ba3b9207: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-412dab7986704f48be75b1fdbdccc9fa received from {username='slave'} at 127.0.0.1:53732
I20260806 20:15:21.782829 30804 tablet_copy_service.cc:433] P d4181d4d907f477f9feddc44ba3b9207: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-412dab7986704f48be75b1fdbdccc9fa on tablet 412dab7986704f48be75b1fdbdccc9fa with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:21.794312 31836 tablet_copy_client.cc:316] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:21.796327 30722 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet 24eb40e43f13412e8f080b6b6ca292de from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:21.796868 30722 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet 24eb40e43f13412e8f080b6b6ca292de from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-24eb40e43f13412e8f080b6b6ca292de
I20260806 20:15:21.803534 30722 tablet_copy_source_session.cc:205] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:21.811250 31836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eb40e43f13412e8f080b6b6ca292de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:21.824263 31836 tablet_copy_client.cc:791] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:21.824793 31836 tablet_copy_client.cc:655] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:21.828059 31836 tablet_copy_client.cc:531] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:21.835916 31836 tablet_bootstrap.cc:492] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:21.859117 31836 tablet_bootstrap.cc:492] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:21.860069 31836 tablet_bootstrap.cc:492] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:21.860703 31836 ts_tablet_manager.cc:1397] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.009s
I20260806 20:15:21.863638 31836 raft_consensus.cc:348] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.864522 31836 raft_consensus.cc:729] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:21.865145 31836 consensus_queue.cc:260] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:21.867537 31836 ts_tablet_manager.cc:1428] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 20:15:21.869067 30722 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-24eb40e43f13412e8f080b6b6ca292de received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:21.869397 30722 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-24eb40e43f13412e8f080b6b6ca292de on tablet 24eb40e43f13412e8f080b6b6ca292de with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:21.931742 31421 raft_consensus.cc:1203] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:22.156358 31421 raft_consensus.cc:1203] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:22.210119 31423 raft_consensus.cc:1203] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:22.353029 31423 raft_consensus.cc:1203] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 20:15:22.382489 31814 raft_consensus.cc:1050] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:22.384552 31814 consensus_queue.cc:237] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:22.393527 31423 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.394541 30788 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.394747 31848 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:22.395928 31814 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:22.396597 30706 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.397778 31785 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:22.408743 31785 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.410473 30787 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.411173 30713 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.427870 30601 catalog_manager.cc:5495] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:22.427939 31423 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.470125 30866 consensus_queue.cc:237] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:22.477540 31423 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:22.479009 31822 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:22.481530 30787 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:22.482831 31822 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:22.485597 31848 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.487164 31421 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.491694 30787 raft_consensus.cc:2932] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.502230 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 88608755c61d42ec8ccc0c3c8e536bd6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:22.501605 30602 catalog_manager.cc:5495] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER 42cabca48db7403b8523a3382a046043 (127.28.186.193) evicted. New cstate: current_term: 1 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.515275 30700 tablet_service.cc:1514] Processing DeleteTablet for tablet 88608755c61d42ec8ccc0c3c8e536bd6 with delete_type TABLET_DATA_TOMBSTONED (TS 42cabca48db7403b8523a3382a046043 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20260806 20:15:22.526096 31857 tablet_replica.cc:330] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043: stopping tablet replica
I20260806 20:15:22.527220 31857 raft_consensus.cc:2229] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:22.528003 31857 raft_consensus.cc:2258] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:22.531829 31857 ts_tablet_manager.cc:1905] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:22.549319 31857 ts_tablet_manager.cc:1918] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:22.549872 31857 log.cc:1196] T 88608755c61d42ec8ccc0c3c8e536bd6 P 42cabca48db7403b8523a3382a046043: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-0-root/wals/88608755c61d42ec8ccc0c3c8e536bd6
I20260806 20:15:22.555341 30589 catalog_manager.cc:4841] TS 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): tablet 88608755c61d42ec8ccc0c3c8e536bd6 (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:22.641260 31828 raft_consensus.cc:1050] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:22.643697 31828 consensus_queue.cc:237] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:22.650969 30866 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.651170 31421 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.651425 30713 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.652460 31828 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:22.653399 31813 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:22.655807 31829 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:22.665745 31847 raft_consensus.cc:1050] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:22.668718 31847 consensus_queue.cc:237] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:22.676980 31828 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.678673 30866 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.737928 30787 raft_consensus.cc:1261] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.745911 31419 raft_consensus.cc:1261] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.748461 31421 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:15:22.782128 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 57962us
I20260806 20:15:22.743018 30599 catalog_manager.cc:5495] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:22.793045 30866 raft_consensus.cc:1261] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.878427 31870 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:22.881224 31868 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:22.883774 31847 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:22.887894 30706 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.895174 30787 consensus_queue.cc:237] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:22.914216 31419 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:22.918448 31867 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:22.919093 31865 raft_consensus.cc:1050] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:22.927613 31865 consensus_queue.cc:237] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:22.935117 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 55014us
I20260806 20:15:22.959664 30711 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:22.961853 31869 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.966866 31421 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.964596 31866 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.972719 30871 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:22.976408 30869 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:22.982947 31865 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:22.993323 31829 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:23.004786 30790 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:23.002198 31420 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 20:15:23.008025 31865 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:23.006598 30711 raft_consensus.cc:2932] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:23.024096 30590 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 412dab7986704f48be75b1fdbdccc9fa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:23.024441 31865 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:23.038652 30792 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:23.048557 31421 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:23.051931 30602 catalog_manager.cc:5495] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New cstate: current_term: 1 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:15:23.059568 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 56361.1us
I20260806 20:15:23.152484 30855 tablet_service.cc:1514] Processing DeleteTablet for tablet 412dab7986704f48be75b1fdbdccc9fa with delete_type TABLET_DATA_TOMBSTONED (TS 58d59beb597f4ed99d84d102ddb41ff5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60882
I20260806 20:15:23.153060 31865 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:23.162065 30866 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:23.125311 30603 catalog_manager.cc:5495] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:23.168012 31882 tablet_replica.cc:330] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5: stopping tablet replica
I20260806 20:15:23.169519 31882 raft_consensus.cc:2229] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:23.170589 31882 raft_consensus.cc:2258] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:23.170948 31420 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:23.174715 31882 ts_tablet_manager.cc:1905] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:23.180445 30785 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:23.197831 30706 consensus_queue.cc:237] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:23.197072 30599 catalog_manager.cc:5495] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 2 to 3, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:23.214851 30788 raft_consensus.cc:1261] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:23.223366 31868 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:23.227358 31420 raft_consensus.cc:1261] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:23.253643 31882 ts_tablet_manager.cc:1918] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:23.254166 31882 log.cc:1196] T 412dab7986704f48be75b1fdbdccc9fa P 58d59beb597f4ed99d84d102ddb41ff5: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/wals/412dab7986704f48be75b1fdbdccc9fa
I20260806 20:15:23.256624 30590 catalog_manager.cc:4841] TS 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): tablet 412dab7986704f48be75b1fdbdccc9fa (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
W20260806 20:15:23.795025 30590 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 538397us
I20260806 20:15:24.352444 31889 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.353564 31889 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.356482 31889 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:24.370963 31889 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.371754 31889 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.376052 31889 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:24.420460 31889 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.421510 31889 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.429311 31889 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:24.461810 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:15:24.484434 31888 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.526904 31888 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.531610 31888 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:24.537889 31888 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.538789 31888 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.541370 31888 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:24.543956 31889 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:24.544656 31889 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.547457 31889 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:24.571642 31886 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.572423 31886 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.595249 31887 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.596062 31887 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.598347 31887 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:24.606854 31888 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:24.633751 31888 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.636101 31888 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:24.636730 31888 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.637275 31888 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.639986 31888 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:24.648916 31813 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.649933 31813 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.652318 31813 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:24.662889 31889 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:24.727376 31878 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.750383 31829 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.760901 31420 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 248) took 1557 ms. Trace:
W20260806 20:15:24.768287 30893 debug-util.cc:398] Leaking SignalData structure 0x7b08001c7080 after lost signal to thread 29422
I20260806 20:15:24.771193 31897 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:24.777007 31892 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:24.781806 31900 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:24.782500 31891 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.782984 31901 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:24.783694 31893 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.798401 31888 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:24.828294 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 805) took 1027 ms. Trace:
I20260806 20:15:24.828651 31886 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:24.829013 31887 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:24.831327 31904 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.832854 31785 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.833806 31871 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.839187 31813 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.804953 31860 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.841209 31898 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.847851 31896 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:24.854028 30710 rpcz_store.cc:276] 0806 20:15:23.800753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:24.827355 (+1026602us) service_pool.cc:227] Handling call
0806 20:15:24.827705 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:24.827879 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:24.827903 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:24.827939 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:24.827965 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:24.828012 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:24.828110 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:24.840049 31889 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.840232 31420 rpcz_store.cc:276] 0806 20:15:23.203797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:23.203937 (+ 140us) service_pool.cc:227] Handling call
0806 20:15:23.223989 (+ 20052us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:23.227783 (+ 3794us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:24.760534 (+1532751us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b54005005d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:24.760765 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40907136}
I20260806 20:15:24.864428 31893 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.878931 31896 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.865201 31887 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.874150 31785 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.883540 31785 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:24.884018 31785 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:24.884552 31785 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.893779 31902 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:24.894529 31902 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } attrs { replace: true } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.898562 31902 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:24.858803 31897 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.899564 31905 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:24.872014 31898 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.854228 31829 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.839892 31878 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.864346 31888 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.900771 31896 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:24.855185 31891 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:25.051523 31906 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:24.876699 31813 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.053285 31785 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:25.053848 31785 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:25.061232 31785 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.063326 31891 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:24.873528 31860 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:25.066781 31829 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.066529 31905 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.850531 31900 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.868290 31901 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:24.864532 31892 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:24.870632 31871 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.131656 31889 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:25.144580 31871 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:25.145324 31909 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:25.067900 31902 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:25.237044 31878 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:25.243276 31900 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:24.872135 31904 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.052218 31906 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.145973 31909 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.247478 31897 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.277791 31785 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:25.264750 31888 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:25.248659 31898 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:25.265419 31893 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:25.266239 31813 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.294907 31887 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.268023 31892 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.286792 31906 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.296366 31860 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:25.267469 31901 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:25.296713 31909 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.286962 31902 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:25.305866 31904 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.329020 31902 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.307518 31905 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.784256 31903 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:25.785004 31903 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.788836 31903 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.821147 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:25.838985 31886 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:25.839648 31886 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:25.841151 31886 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:24.873621 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:25.938452 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 814) took 1065 ms. Trace:
I20260806 20:15:25.938845 30711 rpcz_store.cc:276] 0806 20:15:24.873202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:24.873310 (+ 108us) service_pool.cc:227] Handling call
0806 20:15:25.938359 (+1065049us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:25.968884 31919 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:25.968948 31918 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:25.969765 31919 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.969962 31918 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.972060 31919 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.975651 31918 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.976456 31903 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:25.977128 31903 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.979090 31903 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.979625 31903 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:25.980237 31903 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:25.982197 30893 debug-util.cc:398] Leaking SignalData structure 0x7b08000f07e0 after lost signal to thread 30639
I20260806 20:15:25.982843 31903 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:25.983565 30893 debug-util.cc:398] Leaking SignalData structure 0x7b0800345700 after lost signal to thread 30739
I20260806 20:15:25.983628 31922 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:25.984026 30893 debug-util.cc:398] Leaking SignalData structure 0x7b0800345720 after lost signal to thread 30740
I20260806 20:15:25.984300 31922 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:25.985097 30893 debug-util.cc:398] Leaking SignalData structure 0x7b080010a7c0 after lost signal to thread 30818
I20260806 20:15:25.986459 31922 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.987074 31922 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:25.987603 31922 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:25.989658 31922 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:25.985409 30893 debug-util.cc:398] Leaking SignalData structure 0x7b080028a0c0 after lost signal to thread 30819
W20260806 20:15:25.993041 30893 debug-util.cc:398] Leaking SignalData structure 0x7b080032ca60 after lost signal to thread 30897
W20260806 20:15:25.993137 31388 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.126s user 0.015s sys 0.055s
W20260806 20:15:25.993386 30893 debug-util.cc:398] Leaking SignalData structure 0x7b080032c9c0 after lost signal to thread 30898
W20260806 20:15:25.993657 31388 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.127s user 0.016s sys 0.055s
I20260806 20:15:25.994563 31919 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:25.995087 31919 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:25.997032 31919 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:25.996124 31903 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:25.997699 31919 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:26.000327 31924 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.000849 31924 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.003166 31924 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.003674 31924 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.004240 31924 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.006270 31924 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.006758 31924 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:15:26.007136 30675 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.153s user 0.006s sys 0.026s
W20260806 20:15:26.007608 30675 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.156s user 0.007s sys 0.027s
I20260806 20:15:26.007251 31924 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.008205 31923 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.008914 31920 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.008682 31923 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.009423 31920 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.010815 31923 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.009248 31919 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.011500 31920 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.011464 31919 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.012413 30601 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41560 (request call id 210) took 2760 ms. Trace:
I20260806 20:15:26.012660 31924 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.013352 31919 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.013339 31924 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.014374 31924 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.016390 31924 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:26.016637 30599 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41550 (request call id 229) took 2823 ms. Trace:
I20260806 20:15:26.020416 30711 consensus_queue.cc:237] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.022500 31924 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.023154 31924 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.025318 31924 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.030395 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.038014 31421 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:26.038928 31917 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:26.039588 30674 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.185s user 0.016s sys 0.036s
W20260806 20:15:26.040243 30674 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.191s user 0.016s sys 0.036s
I20260806 20:15:26.040904 31918 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.041394 31918 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.043367 31918 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.017020 30599 rpcz_store.cc:276] 0806 20:15:23.192985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:23.194188 (+ 1203us) service_pool.cc:227] Handling call
0806 20:15:26.016573 (+2822385us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:15:23.210609 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:15:23.211096 (+ 487us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:15:23.211129 (+ 33us) write_op.cc:435] Acquired schema lock
0806 20:15:23.211158 (+ 29us) tablet.cc:660] Decoding operations
0806 20:15:23.223203 (+ 12045us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:15:23.223260 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0806 20:15:23.223286 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0806 20:15:23.223473 (+ 187us) tablet.cc:699] Row locks acquired
0806 20:15:23.223492 (+ 19us) write_op.cc:260] PREPARE: finished
0806 20:15:23.223626 (+ 134us) write_op.cc:270] Start()
0806 20:15:23.223729 (+ 103us) write_op.cc:276] Timestamp: P: 1786047323223580 usec, L: 0
0806 20:15:23.223753 (+ 24us) op_driver.cc:331] REPLICATION: starting
0806 20:15:23.224533 (+ 780us) log.cc:842] Serialized 603 byte log entry
0806 20:15:23.227472 (+ 2939us) op_driver.cc:444] REPLICATION: finished
0806 20:15:23.227841 (+ 369us) write_op.cc:301] APPLY: starting
0806 20:15:23.227917 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0806 20:15:23.231250 (+ 3333us) tablet.cc:1368] finished BulkCheckPresence
0806 20:15:23.231362 (+ 112us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:15:23.231582 (+ 220us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:15:23.231748 (+ 166us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 20:15:23.231806 (+ 58us) write_op.cc:312] APPLY: finished
0806 20:15:23.231995 (+ 189us) log.cc:842] Serialized 23 byte log entry
0806 20:15:23.232145 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:23.232238 (+ 93us) write_op.cc:454] Released schema lock
0806 20:15:23.232278 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0806 20:15:23.232515 (+ 237us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:23.232538 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2211908,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":199,"apply.run_cpu_time_us":1711,"apply.run_wall_time_us":4903,"num_ops":1,"prepare.queue_time_us":135,"prepare.run_cpu_time_us":2337,"prepare.run_wall_time_us":14151,"replication_time_us":3639}]]}
I20260806 20:15:26.046700 31918 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.047281 31918 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.039610 31917 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.050902 31917 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.052699 31923 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.053246 31923 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.056366 31923 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.056905 31923 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.057493 31923 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.058174 31917 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.058763 31917 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.060397 31903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59110 (request call id 1058) took 2872 ms. Trace:
I20260806 20:15:26.060688 31917 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.061522 31923 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.060885 31903 rpcz_store.cc:276] 0806 20:15:23.187609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:23.195753 (+ 8144us) service_pool.cc:227] Handling call
0806 20:15:23.200059 (+ 4306us) log.cc:842] Serialized 508 byte log entry
0806 20:15:26.060312 (+2860253us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2439,"raft.queue_time_us":2406,"raft.run_cpu_time_us":1119,"raft.run_wall_time_us":2519}
I20260806 20:15:26.032918 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.028527 31924 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.062768 31924 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.012784 30601 rpcz_store.cc:276] 0806 20:15:23.251982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:23.780215 (+528233us) service_pool.cc:227] Handling call
0806 20:15:26.012226 (+2232011us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19413,"reported_tablets":1}
I20260806 20:15:26.051472 31918 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.052122 31919 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.069231 31919 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.071346 31919 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.061630 30673 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:26.062232 31923 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.063396 31903 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.073132 31903 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.073635 31919 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.074026 31919 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.074361 31919 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.075143 31903 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.076470 31923 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.079151 31923 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.066100 31931 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.079886 31931 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.084638 31929 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.086288 31929 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.035591 31925 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.082160 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:26.087809 30893 debug-util.cc:398] Leaking SignalData structure 0x7b0800340aa0 after lost signal to thread 31449
W20260806 20:15:26.125509 30893 debug-util.cc:398] Leaking SignalData structure 0x7b0800340a80 after lost signal to thread 31450
I20260806 20:15:26.125716 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.088289 31931 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.090307 31813 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.090783 31909 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.091073 31901 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.130133 31813 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.130291 31901 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.130544 31919 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.132112 31813 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.132346 31901 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.132570 31813 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.133035 31919 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.091593 31887 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.133068 31813 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.130283 31909 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.133735 31887 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.091645 31910 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.136155 31910 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.138300 31910 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.138908 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:26.092101 31898 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.139505 31421 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.139685 31898 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.135483 31813 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.141788 31898 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.092496 31897 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.142483 31897 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.144438 31897 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.092932 31896 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.145043 31896 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.146364 31421 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:26.147022 31896 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.093501 31915 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.147475 30673 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:26.033257 31922 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.147735 31915 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.148406 31922 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:26.148918 30835 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.194:42029 timed out after 1.546s (SENT) [suppressed 3 similar messages]
I20260806 20:15:26.150869 31922 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:26.095952 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.151369 31922 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.151837 31423 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:26.097152 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.098093 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.152734 31418 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.153064 31919 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:26.153939 31919 raft_consensus.cc:686] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:26.093775 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:26.156931 30710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 813) took 1284 ms (client timeout 1687 ms). Trace:
I20260806 20:15:26.100263 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.101778 31876 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:15:26.157312 30710 rpcz_store.cc:269] 0806 20:15:24.872866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:24.872991 (+ 125us) service_pool.cc:227] Handling call
0806 20:15:26.156865 (+1283874us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:26.081002 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 28d6e84eb2874b2b9dca3387b3d98eff with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:26.103173 31895 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:15:26.158838 30589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 77955.1us
I20260806 20:15:26.157922 31419 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:26.158322 31876 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.161175 31876 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:26.104492 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.104980 31886 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.105540 31908 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.106120 31916 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.109329 31860 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.110075 31829 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.119534 30602 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34006 (request call id 120) took 2884 ms. Trace:
I20260806 20:15:26.181367 30602 rpcz_store.cc:276] 0806 20:15:23.235397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:23.237851 (+ 2454us) service_pool.cc:227] Handling call
0806 20:15:26.119452 (+2881601us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1384627,"reported_tablets":2}
I20260806 20:15:26.122324 31924 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.171900 31886 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.128468 31925 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.129748 31929 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.091190 31904 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.135519 31887 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.099021 31877 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.151243 31915 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.069859 30603 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41564 (request call id 220) took 2864 ms. Trace:
W20260806 20:15:26.161677 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 68818.9us
W20260806 20:15:26.165154 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 65593.8us
I20260806 20:15:26.167658 31895 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.168182 31940 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.135645 31909 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.168874 31938 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.169412 31901 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.175686 30673 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:26.122816 31414 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:26.186288 31920 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.151856 31922 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.172016 31908 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.186393 31923 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.190147 30603 rpcz_store.cc:276] 0806 20:15:23.205028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:23.213727 (+ 8699us) service_pool.cc:227] Handling call
0806 20:15:26.069795 (+2856068us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1289606,"reported_tablets":1}
I20260806 20:15:26.194073 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.184943 31829 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.207438 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.192168 31876 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.204803 30754 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.209337 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.212447 30792 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:26.183579 31860 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.189669 31904 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.208544 30869 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.157186 31919 consensus_queue.cc:237] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.216698 31910 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.217115 31926 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.217293 31917 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:26.192057 31940 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.206986 31903 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.220522 31903 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.195749 31938 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.217082 31910 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.225252 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.225929 30792 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:26.222318 31903 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.227792 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.230875 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.230001 30867 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.217151 31895 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.191869 31877 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.226797 31910 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.208747 31901 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.190716 31916 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.211117 31923 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.236414 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.241375 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.242544 31904 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.242731 31940 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.218372 31876 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.215550 31920 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.243851 30792 raft_consensus.cc:1799] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Handling vote request from an unknown peer 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:26.250149 30787 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:26.245007 31886 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.259071 30792 raft_consensus.cc:2396] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 20:15:26.251611 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.252556 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.255033 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.251848 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.261617 30789 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:26.243147 31860 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.242547 31908 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.250445 31938 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.266338 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.252894 31901 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.254122 31829 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.253937 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.234253 31926 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.279211 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.364563 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.364522 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.363919 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.366680 30711 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:26.367251 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:26.363382 30636 debug-util.cc:398] Leaking SignalData structure 0x7b0800095140 after lost signal to thread 29422
I20260806 20:15:26.348109 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.367197 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.370651 30866 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.366230 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.320178 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.257964 31925 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:26.288648 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.382858 31877 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:26.367542 30784 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
W20260806 20:15:26.383837 30636 debug-util.cc:398] Leaking SignalData structure 0x7b0800033640 after lost signal to thread 30639
I20260806 20:15:26.369526 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.366727 31917 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.341733 31922 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.390944 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.367204 31420 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.342518 31933 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:26.402925 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 196581us
I20260806 20:15:26.402513 31933 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.405174 31933 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.377874 31918 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.407128 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.341930 31916 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.407819 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.409811 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.405570 30756 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:26.416190 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.416363 30754 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.420053 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.419629 31936 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:26.422333 31916 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.274333 31932 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.427309 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.425294 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.407546 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:26.423370 30636 debug-util.cc:398] Leaking SignalData structure 0x7b0800320de0 after lost signal to thread 30739
I20260806 20:15:26.426795 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.432369 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.394819 31917 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.434890 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.425634 31916 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.435199 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.435565 31916 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.418808 31923 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.436707 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.424697 31829 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:26.434072 30636 debug-util.cc:398] Leaking SignalData structure 0x7b08002d0360 after lost signal to thread 30740
I20260806 20:15:26.367806 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.451975 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.452672 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.428580 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.450841 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.456050 30787 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:26.451988 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.433388 31876 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:26.418949 31918 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.436733 31920 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.457593 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.427958 31919 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.458364 31420 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.426586 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.459903 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.461277 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.458595 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.463567 31917 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.466408 31388 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:26.481546 31922 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.481855 31926 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.472564 31916 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.482695 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.474246 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.483505 30789 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:26.484189 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.475786 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.485520 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.486109 30784 raft_consensus.cc:1799] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Handling vote request from an unknown peer 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:26.486701 30784 raft_consensus.cc:2396] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 20:15:26.456542 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.478984 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.475034 30754 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.481912 31922 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.492652 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.478842 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.493631 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.493888 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.494771 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.495021 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.495961 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.496063 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.496186 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.496994 30788 raft_consensus.cc:2396] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 20:15:26.497282 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.497731 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.473876 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.498448 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.498739 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.498886 30709 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:26.499635 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.500015 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.501749 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.499558 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.503383 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.505736 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.506742 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.507270 30788 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:26.501710 31422 raft_consensus.cc:1226] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:26.508572 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.509541 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.488222 31919 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:26.473922 30636 debug-util.cc:398] Leaking SignalData structure 0x7b080032d9e0 after lost signal to thread 30818
I20260806 20:15:26.445883 31932 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.510143 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.524014 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.478653 31920 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.484197 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.528265 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.530025 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.510541 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.468436 31910 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.531741 30788 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:26.468041 30783 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:26.517584 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.521483 31388 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:26.523653 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:26.519649 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 44714.8us
I20260806 20:15:26.522089 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.553485 30713 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.523624 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.521061 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:26.514755 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 164363us
I20260806 20:15:26.510216 31389 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:26.523495 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.521682 31829 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.554358 30710 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
W20260806 20:15:26.567396 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 57826.4us
I20260806 20:15:26.573846 31935 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.574486 31935 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.574885 31935 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.559638 31829 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.582525 31829 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.433727 31936 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:26.521335 30636 debug-util.cc:398] Leaking SignalData structure 0x7b08002f7f80 after lost signal to thread 30819
I20260806 20:15:26.568791 31389 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:26.553277 30712 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:26.511881 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.516392 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.512066 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.517869 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.518256 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.523165 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.558468 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.651757 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.651647 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.510995 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.518388 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.646103 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:26.651789 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 83055.8us
I20260806 20:15:26.584784 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.513931 31922 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.512894 31919 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.666198 31389 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:26.658466 31388 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:26.513098 31903 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.516440 31916 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.672328 31936 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:26.673676 31924 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:26.675032 31935 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.676942 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.688750 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.689992 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.688849 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.525317 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.696756 30703 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.699150 31910 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.584483 31929 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:26.675694 31822 consensus_queue.cc:1046] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:26.703331 31822 raft_consensus.cc:3032] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:26.703714 31822 raft_consensus.cc:729] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:26.682525 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.683977 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.636307 31918 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.678812 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.658499 31925 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:26.667037 30791 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:26.659890 30709 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.676512 30708 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.694337 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.692734 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.695741 31903 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.706825 30788 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:26.694044 31924 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.710557 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.712306 30706 raft_consensus.cc:2379] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:26.566080 31920 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:26.709880 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.717460 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.718562 30866 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 20:15:26.725922 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:15:26.712091 30636 debug-util.cc:398] Leaking SignalData structure 0x7b08002fece0 after lost signal to thread 30897
I20260806 20:15:26.720897 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.721045 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.721710 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.719359 30713 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.727751 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.733707 30791 raft_consensus.cc:2932] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:26.734696 31920 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.704648 31822 consensus_queue.cc:260] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.738243 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.739277 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.583227 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.720789 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:26.728168 30636 debug-util.cc:398] Leaking SignalData structure 0x7b080002f680 after lost signal to thread 30898
I20260806 20:15:26.770852 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.726598 31924 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:26.728529 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.785012 31924 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.789084 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.766933 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.790762 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.793149 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.791669 30710 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:26.796525 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.799537 31419 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:26.800773 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.786052 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.785684 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.811177 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.791875 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.832798 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:26.883181 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.814401 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:26.894249 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.812824 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.895902 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:26.805126 30636 debug-util.cc:398] Leaking SignalData structure 0x7b080030e640 after lost signal to thread 31449
W20260806 20:15:26.897758 30636 debug-util.cc:398] Leaking SignalData structure 0x7b0800318aa0 after lost signal to thread 31450
I20260806 20:15:26.898900 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.887782 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.902302 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.902880 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.820744 31916 consensus_queue.cc:1046] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260806 20:15:26.875524 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.922057 31945 raft_consensus.cc:3032] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:26.922493 31945 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:26.903483 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.791715 31822 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.809965 31932 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.805931 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.905350 30756 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:26.813254 31935 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:26.812453 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.896106 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.875296 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.882228 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.836197 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.839321 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.841732 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.842808 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:26.924702 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 266289us
I20260806 20:15:26.935151 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.820870 31922 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.936733 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:26.937184 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.938154 31922 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:26.938407 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:26.939563 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.940399 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.941421 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.942548 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.928712 31916 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:26.901170 31829 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.932250 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.928509 30756 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:26.944896 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:26.945731 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:26.946415 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:26.950898 31916 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:26.952661 30756 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.953419 31916 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.953562 30756 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.949115 30710 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:26.953919 31916 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.960500 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.954559 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:26.958806 30756 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:26.959484 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:26.928583 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 343732us
I20260806 20:15:26.963079 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.934862 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.965775 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.967481 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.966390 30603 catalog_manager.cc:5495] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 2 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:26.967469 30756 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:26.971359 30756 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.972440 30756 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.973752 30756 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.974264 30603 catalog_manager.cc:5495] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 reported cstate change: config changed from index 3 to 4, VOTER 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:26.978183 31910 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.978657 31910 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.979053 31910 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.979496 31415 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:26.944168 31945 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:26.983343 31945 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.961126 31901 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.984902 30756 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.963199 31916 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.002347 31901 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.002755 31901 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:26.973107 30835 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:26.986882 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:26.987835 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:26.990581 31829 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:26.991052 31904 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:26.994163 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.994779 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.984256 31938 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:26.994714 31946 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:26.996013 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.996249 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.997341 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:27.010784 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:27.010918 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.997603 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.998358 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:26.998631 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:27.001943 31421 raft_consensus.cc:1226] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.005533 31860 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.005599 31904 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.008121 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:27.008320 30835 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:27.008888 31936 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.017730 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:27.020360 31860 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:26.970542 31908 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.008704 30756 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:27.030107 30710 raft_consensus.cc:1226] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.025853 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:27.031212 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:27.035171 31938 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.035698 31938 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.025611 31936 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:27.036335 31936 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.036229 31942 consensus_queue.cc:1046] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:27.037473 31942 raft_consensus.cc:3032] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.037825 31942 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.030102 31419 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:27.043270 31877 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.043694 31877 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:27.044049 31877 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.023947 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:27.021580 31829 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.045992 31388 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.021894 31904 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.025766 31860 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.041270 31938 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.048941 31829 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.051949 31922 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.052419 31910 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.065891 31829 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:27.080348 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 69322.3us
I20260806 20:15:27.065655 31919 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
W20260806 20:15:27.082669 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 74438.4us
I20260806 20:15:27.075707 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.074108 31924 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:27.093997 31949 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:15:27.096342 31924 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:27.097282 30757 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:27.071125 31942 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:27.107842 31942 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.110456 30757 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:27.102384 31420 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:27.140990 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 79014.1us
I20260806 20:15:27.105376 31934 raft_consensus.cc:3032] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.155133 31934 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.141273 31956 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.157542 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:27.156409 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.172055 31813 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.180737 31813 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:27.181154 31813 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:27.110915 31822 consensus_queue.cc:1159] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "42cabca48db7403b8523a3382a046043" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20260806 20:15:27.175642 31956 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:27.211412 31956 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.210449 31961 consensus_queue.cc:1046] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:27.189328 31910 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:27.241318 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 67476.1us
I20260806 20:15:27.241505 31388 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:27.255189 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 169154us
I20260806 20:15:27.228992 31934 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:27.244527 31924 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 20:15:27.260656 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 147407us
I20260806 20:15:27.264218 30757 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:27.264850 30835 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:27.265713 30757 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.266891 30757 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.270494 30757 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.274001 31965 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:27.274569 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 196196us
I20260806 20:15:27.267272 31961 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.280035 30835 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.279104 31963 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260806 20:15:27.270860 31960 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.282122 30835 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:27.282603 31958 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.274787 31959 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.275173 31934 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.303893 30864 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:27.277709 31961 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:27.309877 31916 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.312456 31916 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.286751 31960 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:27.292922 31965 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.338012 31910 raft_consensus.cc:3032] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.338832 31813 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.293504 31822 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.330381 30676 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:27.346149 31910 raft_consensus.cc:729] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:27.316318 31961 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.351347 31910 consensus_queue.cc:260] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:27.354789 31904 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.355631 31918 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.359822 30676 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:27.348217 31822 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:15:27.362994 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 305784us
I20260806 20:15:27.343845 31960 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.372713 31813 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.375806 31938 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.377660 31966 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.371697 31822 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.371928 31910 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.366273 31901 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.405884 31936 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.408061 30868 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:27.408355 31936 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:27.392431 30676 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.393738 30756 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:27.372044 31877 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.421954 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.424538 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.423547 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.424058 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.422272 31938 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.432013 30676 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.435122 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.424494 31908 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.436432 31932 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.444765 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:27.443976 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.452188 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.437644 31966 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.455019 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:27.440640 31963 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.456887 31918 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.462167 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.397127 31860 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.465749 31904 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.467675 31901 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.472523 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.475199 31918 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.450431 31932 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.477972 31932 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.479773 30676 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.481977 31963 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.482139 31877 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:27.483626 30676 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.486692 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.488067 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.494338 30676 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.495020 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.496310 30676 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:27.499960 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 208586us
I20260806 20:15:27.500761 30676 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.484007 31966 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.502517 30676 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.485890 31860 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.493794 31903 raft_consensus.cc:2726] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.499289 31926 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 20:15:27.505568 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 235129us
I20260806 20:15:27.494642 31963 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.497152 31917 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.512780 30757 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.502636 30835 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.501395 31920 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.478653 31829 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.503003 31919 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.503810 31966 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.516860 31903 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.512890 31926 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.504581 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.523032 30835 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:27.514972 31860 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.502424 31961 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.526351 30757 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.528803 31955 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.532018 30757 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.523936 31926 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.533407 30757 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.525944 31966 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.535264 31901 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.536396 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:27.537473 30757 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.538859 30757 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.540778 31969 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.542407 30788 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.542793 30792 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.544406 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:27.535375 31860 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.541239 31901 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.547199 30789 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.547322 30786 raft_consensus.cc:1226] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.532960 31970 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.548262 31829 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.548834 30791 raft_consensus.cc:1226] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.533250 31877 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.549727 30757 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.551043 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:27.551447 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.550441 31877 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:27.552902 31877 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.551447 31960 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.556421 31960 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:27.557263 31822 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.558995 31822 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:27.562458 30757 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.561497 31829 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.565864 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.563261 31961 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.566401 31956 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.568600 31956 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.560845 31926 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.567863 30757 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.571540 31877 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.569339 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.578024 30787 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.577924 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:15:27.578739 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 417446us
I20260806 20:15:27.579326 30757 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.581561 30834 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.582574 30757 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.585472 30834 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.586706 30834 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:27.577780 31943 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260806 20:15:27.586980 31964 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.576618 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:27.588938 31964 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.592936 31926 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.580879 31961 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.574146 31910 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.597406 30834 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.604786 31964 raft_consensus.cc:3032] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.606060 31964 raft_consensus.cc:729] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.588243 31934 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.588073 31969 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.606678 31943 consensus_queue.cc:1046] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260806 20:15:27.587809 31956 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.603274 31877 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:27.573194 31932 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.626067 31964 consensus_queue.cc:260] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:27.627722 31964 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.630463 31822 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.600947 31966 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.643383 31960 raft_consensus.cc:3032] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.643826 31960 raft_consensus.cc:729] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.646561 31934 consensus_queue.cc:1046] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:15:27.650597 31955 consensus_queue.cc:1046] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:27.657756 31971 consensus_queue.cc:1046] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260806 20:15:27.659256 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:27.659560 31955 raft_consensus.cc:3032] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.659907 31955 raft_consensus.cc:729] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.658489 31958 consensus_queue.cc:1046] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 20:15:27.660698 31955 consensus_queue.cc:260] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.644749 31960 consensus_queue.cc:260] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.662302 31955 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.662664 31949 raft_consensus.cc:3032] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.640280 31932 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.677758 31972 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:27.678440 31972 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:27.679440 31877 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:27.679920 31936 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:27.682317 31936 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.676964 31960 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.685674 31877 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.690354 31972 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:27.691521 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:27.692215 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:27.694100 30835 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:27.673369 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 76038.2us
I20260806 20:15:27.674054 31971 raft_consensus.cc:3032] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.708847 31971 raft_consensus.cc:729] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.661536 31958 raft_consensus.cc:3032] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.712675 31958 raft_consensus.cc:729] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.663089 31949 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.674768 31910 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.675531 31822 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:27.716573 31822 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:27.718758 31822 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:27.676250 31970 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.697005 31877 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:27.697944 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:27.698875 31966 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:27.697283 31972 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:27.725309 31970 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:27.728591 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:27.734952 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:27.733819 31970 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:27.729068 31936 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:27.711503 30834 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.709939 31971 consensus_queue.cc:260] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.761606 31971 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.713657 31958 consensus_queue.cc:260] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.715687 31910 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:27.721225 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:27.721592 30787 raft_consensus.cc:1226] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.721568 31949 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:27.759320 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 47880.3us
I20260806 20:15:27.782258 30834 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.783915 31946 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59110 (request call id 1066) took 1765 ms. Trace:
I20260806 20:15:27.762890 31964 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:27.792172 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 290062us
I20260806 20:15:27.793041 30676 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.794224 31966 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.795001 30676 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.795871 31966 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.797243 30676 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.797905 30834 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.784406 31946 rpcz_store.cc:276] 0806 20:15:26.018606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:26.018780 (+ 174us) service_pool.cc:227] Handling call
0806 20:15:26.027658 (+ 8878us) log.cc:842] Serialized 508 byte log entry
0806 20:15:27.783804 (+1756146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6606,"raft.run_cpu_time_us":919,"raft.run_wall_time_us":3709,"thread_start_us":35702,"threads_started":1,"wal-append.queue_time_us":20550,"wal-append.run_cpu_time_us":1687,"wal-append.run_wall_time_us":1015904}
I20260806 20:15:27.790774 31955 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.800421 30676 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.799819 31958 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.798792 31966 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.791348 31960 consensus_queue.cc:1046] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260806 20:15:27.801920 30589 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 24eb40e43f13412e8f080b6b6ca292de with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 20:15:27.800809 30834 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.808346 31949 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:27.810266 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 102619us
I20260806 20:15:27.802134 31964 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.812538 30834 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.811115 31955 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.813277 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:27.815659 31966 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.828290 31972 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.829684 30676 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.830407 31966 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.831388 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.831043 31955 raft_consensus.cc:3032] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.832219 31955 raft_consensus.cc:729] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.832331 30832 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.833469 31926 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:27.833873 31926 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:27.834218 31926 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.833596 31972 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.835095 30676 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.835824 31977 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.836228 30676 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.837301 30676 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.838218 31966 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.839152 31966 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.838934 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.835253 31955 consensus_queue.cc:260] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:27.840766 31932 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.840734 31955 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:27.842916 30676 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.843611 31932 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.844600 30676 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.844411 30602 catalog_manager.cc:5495] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New cstate: current_term: 1 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:27.845268 31966 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.845527 31910 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:27.847959 30834 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.848048 30676 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.849228 30676 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.847163 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:27.852010 30864 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.850464 31946 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.859742 30869 raft_consensus.cc:1226] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.861507 30834 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:27.858072 30855 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d6e84eb2874b2b9dca3387b3d98eff with delete_type TABLET_DATA_TOMBSTONED (TS 58d59beb597f4ed99d84d102ddb41ff5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60882
I20260806 20:15:27.852046 31964 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.854507 31932 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.875947 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:27.876423 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:27.878507 30787 raft_consensus.cc:2379] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 2: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:27.879302 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:27.892244 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.894910 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.895852 30869 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:27.896833 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.899523 30757 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.882120 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:27.889935 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:27.890107 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:27.876812 31946 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260806 20:15:27.910017 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.916251 30757 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:27.916500 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.916841 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.917598 30869 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:27.898810 30871 raft_consensus.cc:2932] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:27.917641 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.911209 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.918423 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.919193 30866 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:27.918844 30865 raft_consensus.cc:2454] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:27.920403 30867 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:27.921114 30757 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:27.921285 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:27.885681 31932 consensus_queue.cc:1046] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260806 20:15:27.924706 30706 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.919380 31970 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.926010 30713 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.926390 31956 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.926988 31389 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:27.950142 30792 raft_consensus.cc:1226] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.928998 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.951313 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:27.931367 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.932127 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.937083 30789 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.900997 31910 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.943344 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.957770 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:27.957931 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:27.959515 30707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9"
dest_uuid: "42cabca48db7403b8523a3382a046043"
mode: GRACEFUL
new_leader_uuid: "d4181d4d907f477f9feddc44ba3b9207"
from {username='slave'} at 127.0.0.1:38516
I20260806 20:15:27.959425 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.973847 30866 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:27.920171 31936 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.965199 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.960173 31934 consensus_queue.cc:1046] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260806 20:15:27.980834 30676 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:27.980590 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:27.978958 31971 consensus_queue.cc:1046] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260806 20:15:27.985951 31982 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:27.991030 30868 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:27.991559 31934 raft_consensus.cc:3032] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.991967 31934 raft_consensus.cc:729] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:27.999341 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.976341 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:27.910282 31967 consensus_queue.cc:1046] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260806 20:15:27.986964 30784 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:27.987851 31946 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.006886 30774 tablet_service.cc:1514] Processing DeleteTablet for tablet 24eb40e43f13412e8f080b6b6ca292de with delete_type TABLET_DATA_TOMBSTONED (TS d4181d4d907f477f9feddc44ba3b9207 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33214
I20260806 20:15:28.009042 31967 raft_consensus.cc:3032] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.021936 31967 raft_consensus.cc:729] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:28.023025 31946 raft_consensus.cc:686] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:28.023967 31946 consensus_queue.cc:237] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.022768 31967 consensus_queue.cc:260] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.026301 31967 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.010305 31962 raft_consensus.cc:3032] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:27.992776 31934 consensus_queue.cc:260] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.027015 30861 raft_consensus.cc:1226] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:28.027230 31962 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:28.027529 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:27.998704 31877 raft_consensus.cc:3032] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.028062 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.015087 31970 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:28.018136 30707 raft_consensus.cc:595] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Received request to transfer leadership to TS d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.019153 30787 raft_consensus.cc:1226] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:28.028335 31877 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:28.001493 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:28.031997 30869 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:28.028957 31970 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.047163 30834 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.050369 30713 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:28.051772 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:28.052009 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.054554 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.028205 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.065584 30861 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.066243 31970 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:28.056152 31971 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.064908 31962 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.068588 31962 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.071249 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:28.073824 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:28.075609 31926 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.076787 30708 raft_consensus.cc:1226] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:28.057142 30834 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.050360 31949 consensus_queue.cc:1046] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260806 20:15:28.059934 31934 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.088301 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.060380 31957 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:28.089926 31983 log.cc:924] Time spent T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260806 20:15:28.062803 31877 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.090905 31877 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.092938 30754 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:28.053613 31972 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.059825 31986 consensus_queue.cc:1046] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260806 20:15:28.053283 31985 tablet_replica.cc:330] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: stopping tablet replica
I20260806 20:15:28.045956 30864 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:28.094933 30834 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.098277 30868 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
W20260806 20:15:28.108790 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 181885us
I20260806 20:15:28.109717 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:28.106585 31971 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.115998 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.117259 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.117641 31389 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:28.118330 30866 raft_consensus.cc:2379] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:28.119413 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.119735 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.120009 30867 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:28.106421 31910 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.126433 30676 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.129177 30834 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.128701 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.130247 30869 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:28.130867 31389 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.131980 31389 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.134676 30834 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.126324 31926 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:28.136650 30834 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.121150 31971 raft_consensus.cc:3032] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.127439 31982 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.128233 31957 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.135169 31949 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.147773 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.150310 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.153327 30784 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:28.154148 31985 raft_consensus.cc:2229] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:28.154282 31988 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.154484 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:28.154986 31985 raft_consensus.cc:2258] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:28.140991 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.139078 31958 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.144028 31989 tablet_replica.cc:330] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207: stopping tablet replica
I20260806 20:15:28.162315 31932 consensus_queue.cc:1046] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:28.163753 31949 consensus_queue.cc:1046] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:28.172139 31949 raft_consensus.cc:3032] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.172530 31949 raft_consensus.cc:729] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:28.172621 31932 raft_consensus.cc:3032] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.173034 31932 raft_consensus.cc:729] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:28.173084 30863 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:28.173398 31949 consensus_queue.cc:260] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.173847 31932 consensus_queue.cc:260] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.164610 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.165830 31957 consensus_queue.cc:1046] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:28.176539 31957 raft_consensus.cc:3032] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.176891 31957 raft_consensus.cc:729] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:28.167232 31986 raft_consensus.cc:3032] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.167146 30757 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:28.177634 31957 consensus_queue.cc:260] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.179036 31957 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.179425 31970 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.147346 31987 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:28.181237 31987 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:28.181643 31987 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.183599 31989 raft_consensus.cc:2229] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:15:28.184449 31989 raft_consensus.cc:2258] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:15:28.184623 30861 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:28.185542 30867 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:28.177732 31986 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:28.169561 30603 catalog_manager.cc:5495] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 2 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:28.142925 31918 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.179169 30757 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:28.142818 31936 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.141747 31971 raft_consensus.cc:729] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:28.189765 30757 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:28.202554 30757 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:28.158620 31982 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.158862 31926 consensus_queue.cc:1046] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260806 20:15:28.186769 31986 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.203752 31962 raft_consensus.cc:3032] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.207648 31962 raft_consensus.cc:729] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:28.208757 31932 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.208966 31971 consensus_queue.cc:260] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.210328 31971 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.174886 31949 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.194633 31910 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.212635 31910 raft_consensus.cc:686] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:28.213539 31910 consensus_queue.cc:237] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.195166 30708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4"
dest_uuid: "42cabca48db7403b8523a3382a046043"
mode: GRACEFUL
new_leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
from {username='slave'} at 127.0.0.1:38516
I20260806 20:15:28.216588 30708 raft_consensus.cc:595] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Received request to transfer leadership to TS 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:28.196069 31989 ts_tablet_manager.cc:1905] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:28.198716 31987 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.198173 31985 ts_tablet_manager.cc:1905] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 20:15:28.188776 31988 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.203989 31986 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.203912 31903 consensus_queue.cc:1046] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:28.205385 31956 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.206254 31970 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.207167 30757 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:28.201334 30870 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:28.235275 31970 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.236641 30757 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.237424 31970 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
W20260806 20:15:28.238267 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 106362us
I20260806 20:15:28.238655 30757 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:28.219631 30868 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:28.238883 31389 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.239943 31389 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:28.240016 30757 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.241981 31829 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.242295 31993 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.242883 30708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0"
dest_uuid: "42cabca48db7403b8523a3382a046043"
mode: GRACEFUL
new_leader_uuid: "d4181d4d907f477f9feddc44ba3b9207"
from {username='slave'} at 127.0.0.1:38516
I20260806 20:15:28.243517 30708 raft_consensus.cc:595] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Received request to transfer leadership to TS d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.241099 31982 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.248332 31987 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:28.251595 31389 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.257381 30870 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:28.241158 31961 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.266619 31962 consensus_queue.cc:260] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.267689 31962 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.269515 31987 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.270262 31987 raft_consensus.cc:686] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:28.271121 31987 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.254873 31389 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.282500 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:28.283334 30868 raft_consensus.cc:1789] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.3
I20260806 20:15:28.284886 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:28.286798 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:28.288028 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:28.288457 30869 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:28.289387 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:28.289896 30869 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:28.293601 31829 raft_consensus.cc:686] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:28.294346 31982 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:28.294970 31982 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:28.295540 31982 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.296592 31962 raft_consensus.cc:979] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: : Instructing follower d4181d4d907f477f9feddc44ba3b9207 to start an election
I20260806 20:15:28.294978 30867 raft_consensus.cc:1226] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:28.284318 30868 raft_consensus.cc:2454] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:28.299360 31389 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.299315 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.300603 30868 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 2: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:28.301147 31389 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.296310 31829 consensus_queue.cc:237] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.305699 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:28.297966 31967 raft_consensus.cc:1067] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Signalling peer d4181d4d907f477f9feddc44ba3b9207 to start an election
W20260806 20:15:28.313828 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 72495.2us
I20260806 20:15:28.312209 31962 raft_consensus.cc:979] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: : Instructing follower d4181d4d907f477f9feddc44ba3b9207 to start an election
I20260806 20:15:28.314858 31903 consensus_queue.cc:1046] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:28.315850 31967 raft_consensus.cc:3032] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.316190 31967 raft_consensus.cc:729] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:28.302060 31993 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.317164 31993 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:28.317003 31967 consensus_queue.cc:260] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.317590 31993 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:28.318034 31993 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.302654 31996 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:28.301566 31970 raft_consensus.cc:686] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:28.318729 31967 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.333437 30789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
from {username='slave'} at 127.0.0.1:33236
I20260806 20:15:28.334093 30789 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:15:28.334373 31903 raft_consensus.cc:979] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: : Instructing follower 58f2f6256ee14cb38a1dbcbf5bb590a0 to start an election
I20260806 20:15:28.334414 30789 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.322136 31996 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.323038 31962 raft_consensus.cc:1067] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Signalling peer d4181d4d907f477f9feddc44ba3b9207 to start an election
I20260806 20:15:28.330734 31926 consensus_queue.cc:1046] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260806 20:15:28.337534 31988 raft_consensus.cc:1067] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Signalling peer 58f2f6256ee14cb38a1dbcbf5bb590a0 to start an election
I20260806 20:15:28.369720 31996 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:28.362869 31903 raft_consensus.cc:3032] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.372730 30757 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.373648 32002 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.374302 32002 raft_consensus.cc:686] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:28.373991 30757 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.375561 32002 consensus_queue.cc:237] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.376246 30757 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:28.377550 30757 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.378439 31996 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.378870 30757 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.379062 31961 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.379608 31910 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.379863 31961 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.380111 31910 raft_consensus.cc:686] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:28.380362 31961 raft_consensus.cc:686] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:28.380980 31910 consensus_queue.cc:237] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.378912 31996 raft_consensus.cc:686] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:28.381124 31961 consensus_queue.cc:237] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.386328 31996 consensus_queue.cc:237] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.370570 30789 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:28.379197 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 48907.6us
I20260806 20:15:28.402861 30789 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:28.403041 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:28.403780 30703 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:28.408618 30789 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
from {username='slave'} at 127.0.0.1:33236
I20260806 20:15:28.409178 30789 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:15:28.409439 30789 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.411274 31415 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4"
dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
from {username='slave'} at 127.0.0.1:49046
I20260806 20:15:28.411856 31415 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:15:28.412204 31415 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.417819 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:28.418498 31419 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:28.371486 31903 raft_consensus.cc:729] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:28.432451 31903 consensus_queue.cc:260] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.433828 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:28.434525 30708 raft_consensus.cc:3032] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.434926 30708 raft_consensus.cc:729] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:28.405304 31970 consensus_queue.cc:237] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.435747 30708 consensus_queue.cc:260] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.421926 30756 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:28.440294 30708 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.464296 30789 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:28.458163 32003 log.cc:924] Time spent T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.079s user 0.001s sys 0.001s
I20260806 20:15:28.447956 31956 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:28.466284 30789 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:28.487262 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:28.433434 31982 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:28.541616 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 50911.5us
I20260806 20:15:28.482733 31903 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.471088 31956 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:28.548552 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 157310us
I20260806 20:15:28.549717 31419 raft_consensus.cc:1226] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:28.537185 30601 catalog_manager.cc:5495] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:28.554337 30868 raft_consensus.cc:1261] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:28.563159 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.563879 30867 raft_consensus.cc:2379] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
W20260806 20:15:28.564008 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 263305us
I20260806 20:15:28.553059 31956 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:28.558614 31985 ts_tablet_manager.cc:1918] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 20:15:28.568063 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.571616 31987 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:28.571924 31987 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:28.572201 31987 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.572692 31389 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.574865 31389 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:28.626307 31982 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:28.562419 31926 consensus_queue.cc:1035] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260806 20:15:28.626904 31982 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.627473 31982 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.625300 31993 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.578280 31985 log.cc:1196] T 28d6e84eb2874b2b9dca3387b3d98eff P 58d59beb597f4ed99d84d102ddb41ff5: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-2-root/wals/28d6e84eb2874b2b9dca3387b3d98eff
I20260806 20:15:28.647014 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:28.660755 31993 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:28.661686 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:28.662531 30705 raft_consensus.cc:2379] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:28.663336 30590 catalog_manager.cc:4841] TS 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): tablet 28d6e84eb2874b2b9dca3387b3d98eff (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:28.579802 31918 consensus_queue.cc:1046] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260806 20:15:28.678205 31918 raft_consensus.cc:3032] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:28.638795 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.678591 31918 raft_consensus.cc:729] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:15:28.692127 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 135874us
I20260806 20:15:28.691222 30870 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.714821 30708 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
W20260806 20:15:28.788614 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 69533.6us
I20260806 20:15:28.799306 31987 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.807657 30784 raft_consensus.cc:1261] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:28.809779 31956 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.810217 31415 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.815160 30713 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:28.793148 31989 ts_tablet_manager.cc:1918] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
W20260806 20:15:28.830471 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 47110us
I20260806 20:15:28.833693 30756 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:28.762524 31918 consensus_queue.cc:260] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:28.847092 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 272271us
I20260806 20:15:28.835302 31989 log.cc:1196] T 24eb40e43f13412e8f080b6b6ca292de P d4181d4d907f477f9feddc44ba3b9207: Deleting WAL directory at /tmp/dist-test-taskU5IZJj/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786047231998201-29419-0/minicluster-data/ts-1-root/wals/24eb40e43f13412e8f080b6b6ca292de
I20260806 20:15:28.848984 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.849978 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.851918 30861 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:28.853119 30863 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9"
dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47072
I20260806 20:15:28.865018 30863 raft_consensus.cc:595] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.865430 30863 raft_consensus.cc:603] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:28.854823 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.847010 31961 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.869379 31956 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:28.844360 31987 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:28.875550 32001 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:28.876107 32001 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.878312 32001 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:28.879746 30590 catalog_manager.cc:4841] TS d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029): tablet 24eb40e43f13412e8f080b6b6ca292de (table test-workload [id=9525a22bcf824745aa69b29814e684c4]) successfully deleted
I20260806 20:15:28.881088 31415 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:28.883642 31961 raft_consensus.cc:686] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:28.884459 31961 consensus_queue.cc:237] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:28.855774 30870 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:28.859094 31389 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:28.862023 30867 raft_consensus.cc:1261] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:28.869432 30602 catalog_manager.cc:5495] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 2 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:28.890004 30865 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:28.893640 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:28.900309 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:28.900920 30867 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.901816 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:28.902588 30784 raft_consensus.cc:2379] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 2: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:28.903512 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:28.904168 30708 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:15:28.899111 31993 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.907346 31993 raft_consensus.cc:686] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:28.908267 31993 consensus_queue.cc:237] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.909745 32010 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:28.910244 32010 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.911353 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:28.912036 30703 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:28.912400 32010 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:28.900517 30712 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.913882 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:28.914508 30784 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:28.915570 31389 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:28.916364 31982 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:28.916724 31982 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:28.917026 31982 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:28.894914 31987 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:28.925163 30867 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:28.929831 31982 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:28.934764 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:28.936604 31910 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:28.937211 31910 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:28.937809 31910 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:28.938510 30703 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:28.941840 31389 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:28.942791 31987 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.943254 31987 raft_consensus.cc:686] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:28.944746 30864 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:28.946743 31389 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:28.948499 30756 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:29.023983 32010 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.944559 31987 consensus_queue.cc:237] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.026880 30708 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:29.028322 31829 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:28.923132 32021 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260806 20:15:29.038319 31918 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:29.043886 30603 catalog_manager.cc:5495] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:29.075078 30603 catalog_manager.cc:5495] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:29.047412 30712 raft_consensus.cc:1261] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:29.076900 30603 catalog_manager.cc:5495] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:29.060355 30713 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:29.056751 30703 raft_consensus.cc:1261] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:29.081416 30603 catalog_manager.cc:5495] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
W20260806 20:15:29.097491 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 249716us
I20260806 20:15:29.073026 31982 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:29.073463 32010 raft_consensus.cc:686] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:29.105690 30603 catalog_manager.cc:5495] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:29.107631 30603 catalog_manager.cc:5495] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:29.119398 30712 raft_consensus.cc:1261] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:29.127106 32010 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.092649 31829 raft_consensus.cc:686] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:29.134539 31829 consensus_queue.cc:237] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.137164 31917 consensus_queue.cc:1035] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260806 20:15:29.145784 30705 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:29.093315 31999 consensus_queue.cc:1035] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260806 20:15:29.159273 30756 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:29.165683 31996 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:29.166323 31996 raft_consensus.cc:686] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:29.167227 31996 consensus_queue.cc:237] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.169742 32013 consensus_queue.cc:1035] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:29.172082 32013 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:29.174274 31971 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.174849 31971 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.181941 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:29.182646 30785 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:29.185029 30835 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:29.191692 30867 raft_consensus.cc:1261] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:29.192230 30713 raft_consensus.cc:1261] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:29.141347 31910 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.181051 31971 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:29.188309 31957 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:29.199865 30868 raft_consensus.cc:1261] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:29.200094 31957 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:29.200587 31957 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:29.219704 30867 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:29.226724 32019 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:29.232725 32007 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:29.236660 32007 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.236694 32020 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 20:15:29.245507 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 298851us
I20260806 20:15:29.247308 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:29.248239 30784 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:29.272570 30865 raft_consensus.cc:1261] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:29.215291 32026 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260806 20:15:29.270774 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:29.370700 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:29.378345 30865 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:29.292774 30864 raft_consensus.cc:1261] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:29.370688 31936 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.422079 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:29.422938 30712 raft_consensus.cc:2379] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:29.227397 32019 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.280022 31416 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:29.280169 32022 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.279984 32007 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.421752 31910 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:29.428673 31971 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.293212 32020 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.145s
I20260806 20:15:29.524811 30705 raft_consensus.cc:1261] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:29.526840 32019 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.422765 31936 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.555624 31936 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:29.555814 31962 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.505249 31971 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.556531 31962 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.558269 31971 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:29.558840 31962 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.560911 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:29.561718 30864 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:29.563112 30676 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:29.563961 31962 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:29.564322 31962 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:29.564638 31962 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:29.528802 31957 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.278168 32008 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.207s
I20260806 20:15:29.536499 32025 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:29.568738 32025 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.543519 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:29.437728 32022 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.545760 31946 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:29.578979 31946 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.580928 31946 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.583014 32022 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:29.583698 31957 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:29.578320 32025 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.277634 32032 consensus_queue.cc:1035] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.247s
I20260806 20:15:29.591964 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:29.592840 30864 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:29.594450 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:29.596700 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:29.282615 31970 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.203s
I20260806 20:15:29.597360 30865 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:29.599267 30676 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:29.600173 32025 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:29.600543 32025 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:29.600859 32025 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:29.602191 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:29.607052 31917 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.607966 31917 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.611215 31917 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.612296 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:29.613049 30864 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:29.605530 32022 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:29.614600 32022 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:29.615065 32022 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:29.619287 30676 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:29.620167 32028 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:29.620558 32028 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:29.621032 32028 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:29.633503 32023 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.635545 32023 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.641148 32023 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.656479 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:29.657260 31419 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:15:29.676502 31419 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:29.679301 30832 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:29.667526 31946 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.683372 31946 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.687014 31936 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:29.687604 31936 raft_consensus.cc:686] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:29.690006 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:29.688572 31936 consensus_queue.cc:237] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.691108 30864 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:29.694631 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:29.630884 31416 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
W20260806 20:15:29.695987 30870 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.016s sys 0.066s
I20260806 20:15:29.697336 31926 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:29.716254 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:29.718750 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:29.719673 31419 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:29.715091 31926 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:29.723341 30675 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.857s user 0.046s sys 0.057s
W20260806 20:15:29.726714 30675 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.860s user 0.046s sys 0.057s
W20260806 20:15:29.712117 30870 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.569s user 0.017s sys 0.067s
W20260806 20:15:29.715761 30791 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.007s sys 0.076s
I20260806 20:15:29.698966 32022 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.722733 31415 raft_consensus.cc:1226] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:29.733716 30791 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s user 0.007s sys 0.077s
I20260806 20:15:29.738195 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:29.738867 30712 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:15:29.724299 31946 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.709483 31962 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.727061 31926 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:29.824890 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:29.787364 30712 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:29.838930 30705 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:29.766511 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:29.856003 31419 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
W20260806 20:15:29.735363 31903 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.019s sys 0.082s
I20260806 20:15:29.770735 31920 consensus_queue.cc:1046] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.112s
I20260806 20:15:29.779598 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:29.865597 31920 consensus_queue.cc:1225] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 20:15:29.753070 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:29.866501 30707 raft_consensus.cc:3032] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:29.866844 30707 raft_consensus.cc:729] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:29.867797 30707 consensus_queue.cc:260] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.770907 31971 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:29.864923 32022 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:29.863281 31903 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.729s user 0.020s sys 0.083s
I20260806 20:15:29.753300 31422 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 20:15:29.882689 31829 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.054s sys 0.097s
W20260806 20:15:29.894822 30834 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.017s sys 0.046s
W20260806 20:15:29.908880 30834 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.017s sys 0.046s
I20260806 20:15:29.752974 31918 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.881033 30864 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:29.892233 31996 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260806 20:15:29.839358 30785 consensus_queue.cc:237] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:29.880946 31971 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.880641 32025 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.896064 30707 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:29.910961 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:29.928153 30757 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.020s sys 0.032s
W20260806 20:15:29.905074 30756 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.016s sys 0.036s
W20260806 20:15:29.903950 30708 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s user 0.021s sys 0.042s
I20260806 20:15:29.931942 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:29.925369 31829 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.788s user 0.055s sys 0.098s
W20260806 20:15:29.934345 30757 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.651s user 0.021s sys 0.033s
I20260806 20:15:29.934387 30712 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
W20260806 20:15:29.934849 30756 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.017s sys 0.036s
I20260806 20:15:29.934736 30713 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
W20260806 20:15:29.935851 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 652774us
I20260806 20:15:29.928992 31957 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.928580 32031 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:29.935029 30708 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.021s sys 0.042s
W20260806 20:15:29.962155 30755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.008s sys 0.049s
W20260806 20:15:29.964294 30755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.660s user 0.008s sys 0.049s
I20260806 20:15:29.927031 31918 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.982895 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:29.988771 30713 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:29.985584 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:29.982614 31829 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:29.993377 31829 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.990090 31917 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.995983 31917 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:29.962473 32031 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.996829 31926 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:29.997947 31917 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:29.998454 32031 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.998791 31926 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.999285 31917 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:29.999665 31917 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:29.999974 31917 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:29.964396 31962 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.005707 30711 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:30.005831 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:30.006487 31419 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:30.007640 30673 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:30.007984 30757 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:30.009644 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.010252 30789 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:30.009112 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.013139 30706 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:30.006502 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.015489 30787 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:30.003055 31877 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:29.982733 32023 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:29.995393 31829 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:29.960744 31957 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.020802 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:30.027915 31419 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:30.028043 30790 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.022567 32047 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:30.023221 32019 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:30.023681 32020 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.025692 32046 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:30.026274 32032 consensus_queue.cc:1035] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:30.032258 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.033562 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.049660 30787 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:30.050019 30792 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 2: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:30.034448 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.035804 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.013108 31946 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:30.082484 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:30.047775 32037 log.cc:924] Time spent T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260806 20:15:30.086078 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.039395 32053 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.053599 30705 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:30.038703 31967 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:30.087915 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:30.094898 30792 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 2: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:30.039240 31936 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.052390 31946 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:30.096479 30711 raft_consensus.cc:1261] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:30.054426 31829 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.048842 32046 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:30.098922 32039 log.cc:924] Time spent T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260806 20:15:30.061290 31962 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:30.084156 32044 log.cc:924] Time spent T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20260806 20:15:30.049433 32019 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.053483 32049 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.103488 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:30.105226 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:30.083305 32022 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.129727 30869 raft_consensus.cc:1261] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:30.055696 31996 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260806 20:15:30.037645 32023 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.034066 31877 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.132169 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:30.100940 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 115469us
W20260806 20:15:30.091013 32010 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.010s sys 0.031s
I20260806 20:15:30.138087 30865 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.118733 32025 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:30.146126 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.102486 31946 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:30.135480 32028 consensus_meta.cc:287] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.514s user 0.011s sys 0.000s
I20260806 20:15:30.054692 32020 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.102129 32046 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:30.172104 30865 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:30.174057 30676 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:30.175592 32059 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:30.176105 32059 raft_consensus.cc:686] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:30.182631 31993 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:30.183264 31993 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.184144 30603 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41560 (request call id 215) took 1329 ms. Trace:
I20260806 20:15:30.135910 31971 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:30.185540 31993 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.176924 32059 consensus_queue.cc:237] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.190945 31388 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:30.164816 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:30.192193 31993 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:30.192342 31910 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:30.193162 30603 rpcz_store.cc:276] 0806 20:15:28.854235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:28.854369 (+ 134us) service_pool.cc:227] Handling call
0806 20:15:30.184036 (+1329667us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:15:29.233005 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:15:29.495415 (+262410us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:15:29.495452 (+ 37us) write_op.cc:435] Acquired schema lock
0806 20:15:29.495472 (+ 20us) tablet.cc:660] Decoding operations
0806 20:15:29.496284 (+ 812us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:15:29.496341 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0806 20:15:29.496368 (+ 27us) tablet.cc:683] Acquiring locks for 6 operations
0806 20:15:29.496893 (+ 525us) tablet.cc:699] Row locks acquired
0806 20:15:29.496912 (+ 19us) write_op.cc:260] PREPARE: finished
0806 20:15:29.497076 (+ 164us) write_op.cc:270] Start()
0806 20:15:29.497195 (+ 119us) write_op.cc:276] Timestamp: P: 1786047329497021 usec, L: 0
0806 20:15:29.497212 (+ 17us) op_driver.cc:331] REPLICATION: starting
0806 20:15:29.497890 (+ 678us) log.cc:842] Serialized 2500 byte log entry
0806 20:15:30.024418 (+526528us) op_driver.cc:444] REPLICATION: finished
0806 20:15:30.177942 (+153524us) write_op.cc:301] APPLY: starting
0806 20:15:30.178027 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0806 20:15:30.178225 (+ 198us) tablet.cc:1368] finished BulkCheckPresence
0806 20:15:30.178363 (+ 138us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:15:30.179080 (+ 717us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:15:30.179262 (+ 182us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0806 20:15:30.179302 (+ 40us) write_op.cc:312] APPLY: finished
0806 20:15:30.179551 (+ 249us) log.cc:842] Serialized 53 byte log entry
0806 20:15:30.179706 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:30.179813 (+ 107us) write_op.cc:454] Released schema lock
0806 20:15:30.179851 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0806 20:15:30.180072 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:30.180118 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":111752,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":153224,"apply.run_cpu_time_us":2499,"apply.run_wall_time_us":2500,"num_ops":6,"prepare.queue_time_us":1950,"prepare.run_cpu_time_us":5962,"prepare.run_wall_time_us":309429,"replication_time_us":527082,"thread_start_us":131868,"threads_started":1}]]}
I20260806 20:15:30.198892 31961 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:30.202908 31926 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.203441 31926 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:30.205583 32045 log.cc:924] Time spent T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20260806 20:15:30.166366 32010 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.011s sys 0.031s
I20260806 20:15:30.100100 31967 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.043987 32047 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.205374 31926 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:30.230656 32060 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:30.108352 31829 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.101979 31936 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.136615 31962 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.097210 32053 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.248363 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.251152 30705 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:30.264278 30706 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:30.251322 31994 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20260806 20:15:30.251631 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.252082 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:30.252502 32043 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20260806 20:15:30.253062 30787 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:30.254395 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.272445 30788 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:30.252862 32060 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.256667 32047 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.257104 30863 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:30.257093 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:30.210572 31957 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.274056 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.271342 30707 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
W20260806 20:15:30.275022 31920 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260806 20:15:30.260489 30790 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
W20260806 20:15:30.246805 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 53712.3us
I20260806 20:15:30.260704 30676 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:30.210071 31918 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.262406 30713 raft_consensus.cc:2379] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:30.155751 32022 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.265532 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:30.277966 32050 log.cc:924] Time spent T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.186s user 0.000s sys 0.000s
I20260806 20:15:30.278128 30707 raft_consensus.cc:1261] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:30.279475 30707 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:30.279987 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.281117 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:30.281647 30868 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.257521 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 911) took 1078 ms. Trace:
I20260806 20:15:30.285548 30706 raft_consensus.cc:2354] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:30.268273 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.286381 30869 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:30.269049 30867 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:30.271611 31422 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 2: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:30.273420 32067 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.280085 32053 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.239105 31986 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:30.259475 30835 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:30.234143 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:30.288280 32071 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:30.147641 32049 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.206740 32028 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.289201 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.289369 32023 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.290150 32060 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.288272 32055 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:30.290995 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:30.291951 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:30.292647 32022 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.276641 31917 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:30.322839 32054 log.cc:924] Time spent T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
I20260806 20:15:30.323231 32008 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:30.271435 32013 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260806 20:15:30.324193 31917 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.289434 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.258765 32010 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260806 20:15:30.305068 31422 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:30.305609 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:30.307444 30676 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:30.267390 31993 raft_consensus.cc:686] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:30.285251 30784 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:30.318310 30871 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:30.320091 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 60702.7us
I20260806 20:15:30.322415 30868 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:30.325109 30865 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:30.326596 30866 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:30.354357 30835 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:30.355176 30706 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:30.355638 30756 leader_election.cc:398] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:30.355950 30756 leader_election.cc:401] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:15:30.354907 32030 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
W20260806 20:15:30.356515 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 49141.3us
I20260806 20:15:30.358773 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:30.322311 30754 consensus_peers.cc:487] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ba1969a3911423e8e51fc4f2962515d. This is attempt 1: this message will repeat every 5th retry.
W20260806 20:15:30.360121 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 61190.9us
I20260806 20:15:30.359411 30706 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:30.313846 30791 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.361622 30710 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:15:30.364430 30756 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:30.295339 32020 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:30.324133 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 915) took 1095 ms. Trace:
I20260806 20:15:30.311331 30783 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:30.327214 32060 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:30.326571 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:30.306195 31986 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:30.367610 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 221449us
I20260806 20:15:30.381690 30676 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:30.371454 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.374147 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:30.359331 31917 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:30.362118 31957 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:30.357820 32019 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.399986 30676 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:30.403512 30871 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:30.404320 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.412052 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:30.413378 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.297789 32067 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.386266 31917 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:30.370177 31877 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.414038 30871 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:30.384404 32047 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:30.386157 31957 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:30.452713 32072 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260806 20:15:30.372534 32021 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:30.366000 32055 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.409795 30705 raft_consensus.cc:1226] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:15:30.453112 31957 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.406627 32069 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:30.383101 32060 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:30.459757 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:30.460964 30871 raft_consensus.cc:1261] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:30.371867 32008 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260806 20:15:30.410589 31926 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:30.381325 31993 consensus_queue.cc:237] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.454147 31829 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.455063 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:30.454299 31962 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.456754 31917 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.461808 31936 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:30.470968 32060 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.483424 30871 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:30.406526 32062 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.104s
I20260806 20:15:30.321923 32071 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.480541 30863 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:30.485003 32069 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:30.285717 30708 rpcz_store.cc:276] 0806 20:15:29.179550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:29.180614 (+ 1064us) service_pool.cc:227] Handling call
0806 20:15:29.183582 (+ 2968us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:29.183957 (+ 375us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:29.183988 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:29.215124 (+ 31136us) log.cc:842] Serialized 26 byte log entry
0806 20:15:29.987315 (+772191us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:30.001673 (+ 14358us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:30.010451 (+ 8778us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:30.257261 (+246810us) raft_consensus.cc:1671] finished
0806 20:15:30.257288 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:30.257429 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":177,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":2155,"thread_start_us":690137,"threads_started":1,"wal-append.queue_time_us":784661}
I20260806 20:15:30.507400 32049 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.503842 31946 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.509328 32027 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.532747 32033 raft_consensus.cc:3032] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:30.511461 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.538116 30711 raft_consensus.cc:2354] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3.
I20260806 20:15:30.529191 32069 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.529660 32028 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.518803 32064 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.473724 32021 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.530833 31986 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.544598 32064 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.555408 32055 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.560820 32047 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:30.561348 32047 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:30.567405 31957 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.371966 30703 rpcz_store.cc:276] 0806 20:15:29.228917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:29.233569 (+ 4652us) service_pool.cc:227] Handling call
0806 20:15:29.282196 (+ 48627us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:29.433047 (+150851us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:29.435136 (+ 2089us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:29.604832 (+169696us) log.cc:842] Serialized 26 byte log entry
0806 20:15:30.020852 (+416020us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:30.029515 (+ 8663us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:30.040070 (+ 10555us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:30.323911 (+283841us) raft_consensus.cc:1671] finished
0806 20:15:30.323939 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:30.324078 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5616,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":6755,"thread_start_us":410544,"threads_started":1,"wal-append.queue_time_us":533256}
I20260806 20:15:30.542778 32033 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:30.543627 31962 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:30.593947 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.575919 32067 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.583735 31962 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.600502 30870 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:30.615142 31967 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.602943 32062 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:30.621007 31946 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:30.643844 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 258115us
I20260806 20:15:30.644815 32066 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:30.645555 32066 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.649322 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.649829 32067 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.650321 32067 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.650539 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.651782 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.652348 32067 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:30.652727 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.653311 32067 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:30.653864 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.653872 32067 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.654991 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.656126 32067 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.650621 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.652356 30712 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:30.660315 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.662225 30869 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:30.662533 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:30.672144 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.672966 30869 raft_consensus.cc:2354] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:30.674870 30834 leader_election.cc:398] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3.
I20260806 20:15:30.675254 30834 leader_election.cc:401] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:15:30.608603 32064 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.678500 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.680570 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.681417 30869 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:30.588660 32033 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.682837 30834 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:30.684628 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.685750 30869 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:30.686146 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:30.694005 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 235516us
I20260806 20:15:30.701256 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:30.701894 31422 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.681356 32067 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.726600 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:30.730310 30712 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.702790 31936 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:30.691612 32064 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:30.736748 30861 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:15:30.714083 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:30.729050 32033 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.735910 31936 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:30.726490 32071 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.714252 31993 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260806 20:15:30.683856 32062 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:30.741374 30785 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:15:30.735486 32021 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.728058 32032 consensus_queue.cc:1046] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
W20260806 20:15:30.763923 32046 consensus_meta.cc:287] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.528s user 0.006s sys 0.007s
I20260806 20:15:30.713984 31970 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260806 20:15:30.729714 31917 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.737437 32010 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260806 20:15:30.754457 30870 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:30.783280 30712 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:30.787315 30791 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:30.826939 31917 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.827836 30705 raft_consensus.cc:2379] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:30.764529 31957 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.764850 32019 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:30.755147 32069 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:30.810407 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 408278us
I20260806 20:15:30.813822 31962 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:30.825704 30834 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:30.823890 32032 raft_consensus.cc:3032] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:30.830413 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:30.831568 32092 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 20:15:30.831885 32019 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:30.828212 32093 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260806 20:15:30.853521 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:30.792369 31936 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.742834 32064 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.822899 30785 raft_consensus.cc:595] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:15:30.820153 32049 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:30.793635 32066 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:30.896631 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:30.823496 32046 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:30.903774 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 50348.5us
I20260806 20:15:30.922771 32079 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:30.923353 32079 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:30.923696 32079 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:30.920214 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:30.855239 32089 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:30.894407 31986 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:30.904274 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 49647us
I20260806 20:15:30.859830 32032 raft_consensus.cc:729] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:15:30.862152 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 157442us
I20260806 20:15:30.934649 30790 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:30.935600 30757 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:15:30.857304 31961 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.001s sys 0.017s
W20260806 20:15:30.936673 31961 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.001s sys 0.017s
W20260806 20:15:30.932240 30835 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.015s sys 0.013s
W20260806 20:15:30.938645 30835 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.578s user 0.015s sys 0.013s
W20260806 20:15:30.940996 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 580514us
I20260806 20:15:30.860162 32047 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:30.985085 30867 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.027s sys 0.008s
W20260806 20:15:30.985576 30867 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.028s sys 0.008s
W20260806 20:15:31.010267 30865 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.036s sys 0.060s
W20260806 20:15:31.011165 30865 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.036s sys 0.060s
I20260806 20:15:30.914871 32049 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:30.968600 31970 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:30.967329 32089 raft_consensus.cc:686] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:15:31.044605 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 330438us
I20260806 20:15:30.938978 31986 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:31.406509 30711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.020s sys 0.023s
W20260806 20:15:32.002163 30711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.237s user 0.035s sys 0.030s
I20260806 20:15:31.736364 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:31.065367 31970 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:31.837018 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 66209.5us
I20260806 20:15:31.123898 32062 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:32.076233 32026 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:31.142894 32047 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:32.147317 32026 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.148504 32062 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:31.026057 32031 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:31.010211 31926 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.013s sys 0.025s
W20260806 20:15:31.426548 30755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.703s user 0.015s sys 0.094s
W20260806 20:15:31.980947 30868 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.209s user 0.026s sys 0.014s
W20260806 20:15:31.696585 31419 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.096s user 0.000s sys 0.048s
W20260806 20:15:31.606308 31422 consensus_meta.cc:287] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.904s user 0.003s sys 0.007s
I20260806 20:15:32.004809 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:31.726758 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:31.769140 30871 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.193s user 0.019s sys 0.034s
W20260806 20:15:32.150334 30755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.427s user 0.027s sys 0.114s
W20260806 20:15:32.011582 30866 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 922) took 1865 ms (client timeout 2151 ms). Trace:
I20260806 20:15:31.746433 31389 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:32.178828 31422 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
W20260806 20:15:32.178660 30868 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.407s user 0.026s sys 0.022s
W20260806 20:15:32.178145 30871 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.602s user 0.024s sys 0.041s
W20260806 20:15:32.178478 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 451805us
W20260806 20:15:32.178540 30866 rpcz_store.cc:269] 0806 20:15:30.130443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.131925 (+ 1482us) service_pool.cc:227] Handling call
0806 20:15:31.995632 (+1863707us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54000852d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:31.995759 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":394365824}
W20260806 20:15:32.177556 31419 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.577s user 0.001s sys 0.066s
W20260806 20:15:31.537685 32085 log.cc:924] Time spent T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
W20260806 20:15:32.186769 31422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49046 (request call id 337) took 1478 ms (client timeout 1714 ms). Trace:
W20260806 20:15:31.835433 31994 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.014s user 0.013s sys 0.022s
I20260806 20:15:32.146100 31970 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:31.845410 32079 consensus_meta.cc:287] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.921s user 0.001s sys 0.006s
W20260806 20:15:32.149906 31926 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.668s user 0.026s sys 0.049s
W20260806 20:15:31.845268 32033 consensus_meta.cc:287] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.085s user 0.001s sys 0.007s
W20260806 20:15:31.886958 32060 consensus_meta.cc:287] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.384s user 0.011s sys 0.002s
I20260806 20:15:32.189574 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:32.185571 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 940) took 2036 ms. Trace:
W20260806 20:15:31.845860 32008 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.004s sys 0.004s
W20260806 20:15:31.724466 32067 consensus_meta.cc:287] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.998s user 0.005s sys 0.004s
W20260806 20:15:32.190169 31422 rpcz_store.cc:269] 0806 20:15:30.700909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.701021 (+ 112us) service_pool.cc:227] Handling call
0806 20:15:32.179288 (+1478267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:32.182783 30676 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:32.147193 31986 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:31.581338 32027 consensus_meta.cc:287] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.036s user 0.009s sys 0.002s
I20260806 20:15:32.192664 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 942) took 1827 ms. Trace:
I20260806 20:15:31.044359 32032 consensus_queue.cc:260] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:32.078403 32088 log.cc:924] Time spent T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
W20260806 20:15:31.451385 30708 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.014s sys 0.038s
I20260806 20:15:31.908818 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:31.539274 32049 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:32.148614 32047 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260806 20:15:32.062371 32078 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:31.538300 32046 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:32.045491 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 934) took 1772 ms. Trace:
I20260806 20:15:32.024873 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 946) took 1327 ms. Trace:
I20260806 20:15:32.077064 32101 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:31.518543 31962 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:32.195824 31994 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.370s user 0.020s sys 0.028s
I20260806 20:15:31.992718 32102 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:32.182181 31903 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 20:15:31.746104 32030 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.833s user 0.007s sys 0.022s
I20260806 20:15:32.034859 32086 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:31.426287 31946 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:32.149057 32062 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.038448 32080 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:31.969964 32059 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:32.083294 32105 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:31.811152 31877 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:32.060608 31829 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:32.196580 30757 leader_election.cc:398] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:31.842042 32048 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.497s
W20260806 20:15:32.285224 30866 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1169) took 1364 ms (client timeout 1731 ms). Trace:
I20260806 20:15:32.282495 30871 rpcz_store.cc:276] 0806 20:15:30.144893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.511562 (+366669us) service_pool.cc:227] Handling call
0806 20:15:30.511844 (+ 282us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.515450 (+ 3606us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.515481 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:30.515908 (+ 427us) log.cc:842] Serialized 26 byte log entry
0806 20:15:32.180612 (+1664704us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:32.180637 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.180682 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:32.180714 (+ 32us) raft_consensus.cc:1671] finished
0806 20:15:32.180720 (+ 6us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:32.180799 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3295,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":13598,"thread_start_us":1604585,"threads_started":1,"wal-append.queue_time_us":1513633}
I20260806 20:15:32.283689 32079 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.296010 30713 rpcz_store.cc:276] 0806 20:15:30.365076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.370384 (+ 5308us) service_pool.cc:227] Handling call
0806 20:15:32.191986 (+1821602us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:32.192190 (+ 204us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:32.192365 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:32.192391 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:32.192427 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:32.192454 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.192501 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:32.192606 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5590272}
I20260806 20:15:31.610237 32064 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:31.537916 32089 consensus_queue.cc:237] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:32.352938 30833 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.013s sys 0.014s
W20260806 20:15:32.353772 30834 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.000s sys 0.027s
I20260806 20:15:32.271443 32026 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:32.285025 32008 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.950s user 0.008s sys 0.013s
I20260806 20:15:32.279681 32022 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:32.279320 31926 raft_consensus.cc:686] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:32.295200 32060 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.720814 31970 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.667443 32010 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:32.724123 32010 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.662855 30602 catalog_manager.cc:5495] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 2 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:32.724992 31970 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:32.427878 32116 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260806 20:15:32.599135 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1142) took 2190 ms. Trace:
I20260806 20:15:32.726167 31962 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.620656 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 925) took 2467 ms. Trace:
I20260806 20:15:32.727303 31903 raft_consensus.cc:686] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:15:32.697095 30755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.065s sys 0.078s
I20260806 20:15:32.727497 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 941) took 2363 ms. Trace:
I20260806 20:15:32.662395 32072 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:15:32.434741 32115 log.cc:924] Time spent T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
I20260806 20:15:32.664110 32066 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:32.728133 30707 rpcz_store.cc:276] 0806 20:15:30.364721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.365978 (+ 1257us) service_pool.cc:227] Handling call
0806 20:15:30.405942 (+ 39964us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.938215 (+532273us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.941166 (+ 2951us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:31.009630 (+ 68464us) log.cc:842] Serialized 26 byte log entry
0806 20:15:31.959814 (+950184us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:31.965627 (+ 5813us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:31.987855 (+ 22228us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:32.727271 (+739416us) raft_consensus.cc:1671] finished
0806 20:15:32.727303 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:32.727439 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3484,"raft.queue_time_us":33761,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":3669,"thread_start_us":685690,"threads_started":2,"wal-append.queue_time_us":1141072}
I20260806 20:15:32.730260 30864 raft_consensus.cc:2354] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3.
I20260806 20:15:32.723037 32102 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:32.731720 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 536569us
I20260806 20:15:32.732580 32102 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:32.732766 30712 raft_consensus.cc:2354] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:32.732923 30757 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:32.354779 30708 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.637s user 0.028s sys 0.052s
W20260806 20:15:32.634824 32107 log.cc:924] Time spent T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.179s user 0.001s sys 0.000s
I20260806 20:15:32.324576 32027 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:32.364645 32078 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.733959 32066 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.735086 32059 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.663604 32023 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:32.599315 32124 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:32.736160 32023 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.736565 32124 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:32.596459 30832 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.196:35519 timed out after 1.507s (SENT) [suppressed 13 similar messages]
I20260806 20:15:32.562491 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 35) took 2556 ms. Trace:
I20260806 20:15:32.599785 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 922) took 2641 ms. Trace:
I20260806 20:15:32.646819 32103 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:15:32.739144 30755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.534s user 0.068s sys 0.080s
W20260806 20:15:32.728693 32030 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.940s user 0.021s sys 0.052s
I20260806 20:15:32.669407 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:32.354430 32067 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3.
W20260806 20:15:32.448249 31936 consensus_meta.cc:287] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.518s user 0.012s sys 0.000s
I20260806 20:15:32.740216 30710 rpcz_store.cc:276] 0806 20:15:29.958253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:29.959519 (+ 1266us) service_pool.cc:227] Handling call
0806 20:15:30.040066 (+ 80547us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:30.040248 (+ 182us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.366355 (+326107us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.599392 (+2233037us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54001b6054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:32.599697 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":734264832}
W20260806 20:15:32.741335 30834 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.006s user 0.000s sys 0.028s
W20260806 20:15:32.741586 30866 rpcz_store.cc:269] 0806 20:15:30.921254 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:32.189256 (+1268002us) service_pool.cc:227] Handling call
0806 20:15:32.285139 (+ 95883us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:32.720737 31421 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 221) took 1607 ms. Trace:
I20260806 20:15:32.740849 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:32.741133 30833 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.990s user 0.014s sys 0.014s
I20260806 20:15:32.545308 32120 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:32.724454 32022 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.738510 30602 catalog_manager.cc:5495] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 2 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 20:15:32.746443 31389 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.016s sys 0.029s
I20260806 20:15:32.729038 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:32.758973 30871 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
W20260806 20:15:32.769407 30871 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1166) took 1850 ms (client timeout 1985 ms). Trace:
W20260806 20:15:32.769829 30871 rpcz_store.cc:269] 0806 20:15:30.918539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:32.729714 (+1811175us) service_pool.cc:227] Handling call
0806 20:15:32.769338 (+ 39624us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:32.769428 30602 catalog_manager.cc:5495] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 2 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:32.771173 30713 raft_consensus.cc:2379] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:32.773527 30757 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:32.758251 31421 rpcz_store.cc:276] 0806 20:15:31.113289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:31.114037 (+ 748us) service_pool.cc:227] Handling call
0806 20:15:31.997802 (+883765us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:31.998007 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:15:32.242630 (+244623us) log.cc:842] Serialized 14 byte log entry
0806 20:15:32.720026 (+477396us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:32.720381 (+ 355us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:32.720427 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.720502 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:32.720666 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1511,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":110,"thread_start_us":588893,"threads_started":2}
W20260806 20:15:32.775498 31389 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.016s sys 0.029s
W20260806 20:15:32.776152 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 1.02976e+06us
W20260806 20:15:32.776604 30756 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.000s sys 0.002s
W20260806 20:15:32.776929 30756 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.507s user 0.000s sys 0.002s
I20260806 20:15:32.352568 31986 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:32.662803 32033 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:32.731110 32079 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:32.725378 32064 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:32.727541 30867 rpcz_store.cc:276] 0806 20:15:30.133518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.299129 (+165611us) service_pool.cc:227] Handling call
0806 20:15:30.319987 (+ 20858us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.326699 (+ 6712us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.326726 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:30.327100 (+ 374us) log.cc:842] Serialized 26 byte log entry
0806 20:15:30.985940 (+658840us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:30.985982 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:30.986081 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:32.600472 (+1614391us) raft_consensus.cc:1671] finished
0806 20:15:32.600493 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:32.600612 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6342,"raft.queue_time_us":6450,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":4199,"thread_start_us":616172,"threads_started":1,"wal-append.queue_time_us":1479773}
W20260806 20:15:32.620805 32113 log.cc:924] Time spent T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.235s user 0.000s sys 0.000s
W20260806 20:15:32.722659 31387 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.886s user 0.000s sys 0.026s
I20260806 20:15:32.726890 30790 rpcz_store.cc:276] 0806 20:15:30.373741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.373888 (+ 147us) service_pool.cc:227] Handling call
0806 20:15:32.564087 (+2190199us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b540014a5d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:32.564219 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1266505600}
I20260806 20:15:32.736893 32066 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:32.654202 32108 log.cc:924] Time spent T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260806 20:15:32.380223 32101 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.729506 31946 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.726323 32086 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.739975 31926 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.728201 31903 consensus_queue.cc:237] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.976351 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:32.905328 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:33.041848 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:33.018926 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 242004us
I20260806 20:15:32.904876 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:32.847963 30675 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.000s sys 0.019s
W20260806 20:15:32.847761 30676 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.000s sys 0.015s
W20260806 20:15:33.092064 30676 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.809s user 0.000s sys 0.020s
I20260806 20:15:32.731122 32060 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:33.092767 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 910734us
I20260806 20:15:33.093601 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:33.044113 30675 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.766s user 0.000s sys 0.022s
I20260806 20:15:32.734282 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 944) took 2279 ms. Trace:
W20260806 20:15:32.965153 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 116635us
I20260806 20:15:33.095818 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:33.088194 30713 raft_consensus.cc:2396] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:32.726327 32010 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:32.807336 30757 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:32.887121 30869 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:33.088135 30871 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:32.947868 32136 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:32.738698 32023 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:33.088745 32064 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.888886 32026 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:32.912699 32013 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:32.945071 32048 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:32.754128 31877 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.887871 32046 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:32.767992 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:33.139783 31994 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:32.728521 31962 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:32.967856 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 371426us
I20260806 20:15:33.196388 32122 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:33.159595 32030 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:32.738758 32124 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:33.148968 30835 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.003s sys 0.011s
I20260806 20:15:32.758769 32102 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:33.153071 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 59070us
W20260806 20:15:33.016614 32118 log.cc:924] Time spent T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.296s user 0.000s sys 0.001s
I20260806 20:15:32.734236 32080 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.981765 32055 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:33.190800 32023 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:32.800063 32027 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:33.140878 31387 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.306s user 0.000s sys 0.034s
I20260806 20:15:32.947674 32130 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:33.089393 32138 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:32.912595 32047 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:32.733738 31829 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.775736 30870 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
W20260806 20:15:33.241160 30870 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1170) took 2319 ms (client timeout 1590 ms). Trace:
W20260806 20:15:33.174333 31386 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.017s sys 0.035s
W20260806 20:15:33.175021 30867 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1171) took 2253 ms (client timeout 2047 ms). Trace:
W20260806 20:15:33.155901 31416 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.010s sys 0.004s
I20260806 20:15:32.736913 32078 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:32.945906 32123 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:32.947872 32129 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:32.758265 31936 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:32.781224 32103 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.812469 32007 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.366s
I20260806 20:15:33.159129 32060 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:33.171954 32026 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:33.205984 32149 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:33.204599 31994 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:33.206575 31419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48918 (request call id 252) took 2605 ms. Trace:
I20260806 20:15:32.812582 32033 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:33.146507 31986 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:33.186897 32010 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:32.753589 32105 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:33.174467 32048 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:33.204473 32030 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:33.197043 32122 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:32.790937 32120 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:33.180684 32013 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:33.189991 32046 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:33.159296 32136 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:33.105419 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:15:33.157523 30711 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.005s sys 0.009s
I20260806 20:15:33.166222 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:33.140504 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 45574.2us
W20260806 20:15:33.141461 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 334194us
I20260806 20:15:32.994207 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:33.188683 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:33.177145 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:33.207415 30835 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.585s user 0.003s sys 0.011s
W20260806 20:15:33.513288 30834 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.043s sys 0.087s
W20260806 20:15:33.242319 30867 rpcz_store.cc:269] 0806 20:15:30.921778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.091808 (+2170030us) service_pool.cc:214] Skipping call since client already timed out
0806 20:15:33.174939 (+ 83131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:15:33.211346 30863 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
W20260806 20:15:33.517273 30867 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1167) took 2598 ms (client timeout 2063 ms). Trace:
W20260806 20:15:33.517815 30867 rpcz_store.cc:269] 0806 20:15:30.918807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.516712 (+2597905us) service_pool.cc:214] Skipping call since client already timed out
0806 20:15:33.516951 (+ 239us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:15:33.517871 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:32.741153 30784 rpcz_store.cc:276] 0806 20:15:30.005707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.012387 (+ 6680us) service_pool.cc:227] Handling call
0806 20:15:30.033342 (+ 20955us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:30.354079 (+320737us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.562123 (+2208044us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54000131d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:32.562387 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":515114240}
W20260806 20:15:33.519263 30867 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1163) took 2601 ms (client timeout 2082 ms). Trace:
W20260806 20:15:33.519784 30864 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1142) took 3417 ms (client timeout 2228 ms). Trace:
W20260806 20:15:33.519793 30867 rpcz_store.cc:269] 0806 20:15:30.917774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.519002 (+2601228us) service_pool.cc:214] Skipping call since client already timed out
0806 20:15:33.519173 (+ 171us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:15:33.213830 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 928) took 3079 ms. Trace:
W20260806 20:15:33.140358 30871 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 56) took 2200 ms (client timeout 2014 ms). Trace:
I20260806 20:15:33.645149 31917 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:32.363365 30703 rpcz_store.cc:276] 0806 20:15:30.698108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.698655 (+ 547us) service_pool.cc:227] Handling call
0806 20:15:30.699179 (+ 524us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.699526 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.699556 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:30.700595 (+ 1039us) log.cc:842] Serialized 513 byte log entry
0806 20:15:30.927159 (+226564us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:30.931690 (+ 4531us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:30.938018 (+ 6328us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:31.988996 (+1050978us) raft_consensus.cc:1671] finished
0806 20:15:31.989025 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:32.010803 (+ 21778us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":209726,"threads_started":1,"wal-append.queue_time_us":1140709,"wal-append.run_cpu_time_us":16346,"wal-append.run_wall_time_us":1209883}
I20260806 20:15:33.645902 31917 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:33.646462 31917 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:33.646703 32022 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:33.518151 30863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 57) took 2575 ms (client timeout 1665 ms). Trace:
I20260806 20:15:33.504410 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:33.501937 30705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 943) took 3117 ms (client timeout 2438 ms). Trace:
W20260806 20:15:33.518297 30834 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.712s user 0.045s sys 0.087s
I20260806 20:15:33.506625 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:15:33.478472 30712 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 954) took 1480 ms (client timeout 1602 ms). Trace:
I20260806 20:15:33.518290 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 927) took 3384 ms. Trace:
I20260806 20:15:33.626863 32132 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:33.493853 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:33.505705 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:15:33.463649 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 468924us
W20260806 20:15:33.448906 30602 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.010s sys 0.013s
W20260806 20:15:33.459030 30707 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.011s sys 0.055s
I20260806 20:15:33.496549 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:33.242213 31829 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:33.637976 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 532465us
W20260806 20:15:33.520414 30864 rpcz_store.cc:269] 0806 20:15:30.101746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.106890 (+ 5144us) service_pool.cc:227] Handling call
0806 20:15:30.188453 (+ 81563us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:30.319265 (+130812us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71bb7c 00007f714e71b73b 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:33.519182 (+3199917us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b54002a1054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:33.519521 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5471415936}
W20260806 20:15:33.384123 32144 log.cc:924] Time spent T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260806 20:15:33.647140 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:33.628140 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:33.668294 30713 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:33.669531 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:33.669605 32026 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:33.494395 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:33.242286 31386 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.048s user 0.017s sys 0.036s
I20260806 20:15:33.229665 32027 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:33.241616 30870 rpcz_store.cc:269] 0806 20:15:30.921513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:31.432829 (+511316us) service_pool.cc:227] Handling call
0806 20:15:32.669208 (+1236379us) spinlock_profiling.cc:243] Waited 5.36 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:33.241069 (+571861us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12329216}
W20260806 20:15:33.645399 30871 rpcz_store.cc:269] 0806 20:15:30.940128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:32.770577 (+1830449us) service_pool.cc:227] Handling call
0806 20:15:33.140288 (+369711us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:33.658511 30712 rpcz_store.cc:269] 0806 20:15:31.998301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:32.004521 (+ 6220us) service_pool.cc:227] Handling call
0806 20:15:33.477874 (+1473353us) spinlock_profiling.cc:243] Waited 555 ms on lock 0x7b54001b5654. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:33.478215 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1275529472}
W20260806 20:15:33.382685 32090 log.cc:924] Time spent T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 1.390s user 0.007s sys 0.000s
I20260806 20:15:33.468228 30603 catalog_manager.cc:5495] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:33.316834 31946 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:33.667032 30602 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.756s user 0.020s sys 0.032s
I20260806 20:15:33.222297 32062 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:33.296629 31877 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:33.217922 32102 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:33.673921 30707 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.935s user 0.015s sys 0.062s
I20260806 20:15:33.658337 30865 rpcz_store.cc:276] 0806 20:15:30.134082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.357649 (+223567us) service_pool.cc:227] Handling call
0806 20:15:30.357972 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:31.011514 (+653542us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:31.011938 (+ 424us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:31.768391 (+756453us) log.cc:842] Serialized 26 byte log entry
0806 20:15:32.194102 (+425711us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:32.194140 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.194214 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:33.517741 (+1323527us) raft_consensus.cc:1671] finished
0806 20:15:33.517783 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:33.518065 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2130,"raft.queue_time_us":22809,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":5432,"thread_start_us":1023399,"threads_started":2,"wal-append.queue_time_us":749296}
I20260806 20:15:33.645375 31419 rpcz_store.cc:276] 0806 20:15:30.594720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.594892 (+ 172us) service_pool.cc:227] Handling call
0806 20:15:30.595246 (+ 354us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.595586 (+ 340us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.595612 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:30.600060 (+ 4448us) log.cc:842] Serialized 26 byte log entry
0806 20:15:32.184894 (+1584834us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:32.184935 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.185678 (+ 743us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:33.200062 (+1014384us) raft_consensus.cc:1671] finished
0806 20:15:33.200084 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:33.200232 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":87,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":3856,"thread_start_us":1584400,"threads_started":1,"wal-append.queue_time_us":1682134}
W20260806 20:15:34.254105 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 82355.7us
W20260806 20:15:34.254570 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 417246us
I20260806 20:15:33.317102 32064 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.272195 32059 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.273782 31877 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:34.274824 31877 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.278370 32032 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:34.278915 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 262112us
I20260806 20:15:34.279260 32105 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:33.719414 32123 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.281188 32120 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.281910 30835 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:33.245971 32047 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:33.315701 32019 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:33.305204 32127 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:33.304760 32119 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:33.303292 32128 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:33.803131 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 172243us
I20260806 20:15:33.653954 32031 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:33.728008 31986 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:33.256479 32138 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:33.836710 32078 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:34.286724 32013 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:33.839118 30861 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
W20260806 20:15:34.288152 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 451844us
I20260806 20:15:34.287015 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:34.288834 32067 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:34.278549 31421 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 346) took 1110 ms. Trace:
I20260806 20:15:33.316442 32101 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.290968 32120 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:34.290473 30603 catalog_manager.cc:5495] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
W20260806 20:15:33.728135 30711 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.279s user 0.010s sys 0.011s
I20260806 20:15:33.771617 32150 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:33.725225 32148 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:33.649734 30863 rpcz_store.cc:269] 0806 20:15:30.942466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:31.908463 (+965997us) service_pool.cc:227] Handling call
0806 20:15:33.517656 (+1609193us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b5400376754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:33.517899 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1950813056}
W20260806 20:15:33.728801 31416 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 1.242s user 0.011s sys 0.010s
I20260806 20:15:34.311009 32102 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.292804 31877 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:33.728590 32060 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.293140 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:33.658529 30705 rpcz_store.cc:269] 0806 20:15:30.384430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.499868 (+115438us) service_pool.cc:227] Handling call
0806 20:15:33.462917 (+2963049us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b540039eed4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:33.501695 (+ 38778us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":651834752}
I20260806 20:15:33.727787 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.238319 32061 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:34.294167 32046 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.294945 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:33.141412 30708 rpcz_store.cc:276] 0806 20:15:30.455025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.534539 (+ 79514us) service_pool.cc:227] Handling call
0806 20:15:30.554625 (+ 20086us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.586135 (+ 31510us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.587249 (+ 1114us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:30.611018 (+ 23769us) log.cc:842] Serialized 26 byte log entry
0806 20:15:32.733893 (+2122875us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:32.733939 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.734012 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:32.734059 (+ 47us) raft_consensus.cc:1671] finished
0806 20:15:32.734073 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:32.734222 (+ 149us) inbound_call.cc:173] Queueing success response
0806 20:15:32.954717 (+220495us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b48001c0158. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71565167f1 00007f715650d8e7 00007f715650d77a 00007f715660195f 00007f7156601627 00007f7156615b2c 00007f7156615ad8 00007f7156615a58 00007f7156615a13 00007f7156614cbc 00007f7155596be6
Metrics: {"raft.queue_time_us":93,"raft.run_cpu_time_us":49501,"raft.run_wall_time_us":2388063,"spinlock_wait_cycles":3596544,"thread_start_us":2018908,"threads_started":1,"wal-append.queue_time_us":1439038,"wal-append.run_cpu_time_us":12204,"wal-append.run_wall_time_us":1032917}
I20260806 20:15:32.363030 30709 rpcz_store.cc:276] 0806 20:15:30.254750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.280783 (+ 26033us) service_pool.cc:227] Handling call
0806 20:15:30.281091 (+ 308us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.393530 (+112439us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.405640 (+ 12110us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:30.479966 (+ 74326us) log.cc:842] Serialized 26 byte log entry
0806 20:15:30.991916 (+511950us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:31.013016 (+ 21100us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:31.036717 (+ 23701us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:31.995868 (+959151us) raft_consensus.cc:1671] finished
0806 20:15:32.002298 (+ 6430us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:32.026627 (+ 24329us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1898,"raft.queue_time_us":7304,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":70,"thread_start_us":524820,"threads_started":2,"wal-append.queue_time_us":1313183,"wal-append.run_cpu_time_us":2121,"wal-append.run_wall_time_us":1101768}
I20260806 20:15:33.295966 32135 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:33.728600 30867 raft_consensus.cc:1261] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:33.727993 32030 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:34.278671 32136 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.298075 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:33.866323 32149 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.029888 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.077306 32080 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.299553 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:33.887341 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:33.894999 32048 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.300388 32027 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:33.877357 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.095996 30703 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:34.045143 32142 ts_tablet_manager.cc:927] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.025657 32103 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:33.866848 32010 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:33.770023 30833 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.004s sys 0.021s
I20260806 20:15:33.790328 30710 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:34.290577 31936 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:33.875564 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:33.725687 30788 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
W20260806 20:15:33.725613 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1165) took 2728 ms (client timeout 1714 ms). Trace:
I20260806 20:15:33.255023 32131 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:34.344099 30833 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.553s user 0.004s sys 0.033s
W20260806 20:15:34.253933 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 132289us
I20260806 20:15:34.293169 32123 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.302238 32049 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:34.310714 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:33.316108 32086 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:34.345814 30788 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1151) took 1324 ms (client timeout 1731 ms). Trace:
I20260806 20:15:34.210431 32129 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.311079 31994 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:34.333348 30861 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1161) took 3416 ms (client timeout 1775 ms). Trace:
I20260806 20:15:34.310597 32127 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.338126 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:34.305583 32089 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.509s user 0.008s sys 0.005s
I20260806 20:15:34.172875 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:33.866536 32074 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:34.097277 31971 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:34.355074 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1328) took 1161 ms. Trace:
I20260806 20:15:34.355347 31971 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.356130 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.357131 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53732 (request call id 36) took 1163 ms. Trace:
I20260806 20:15:34.357254 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 940) took 3993 ms. Trace:
I20260806 20:15:34.355096 32074 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.357542 30784 rpcz_store.cc:276] 0806 20:15:33.194184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.870953 (+676769us) service_pool.cc:227] Handling call
0806 20:15:34.320417 (+449464us) spinlock_profiling.cc:243] Waited 7.83 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.357066 (+ 36649us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18015488}
I20260806 20:15:34.358321 31971 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.358958 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:34.359694 30784 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:34.360129 32131 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.363071 32131 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.359478 30706 rpcz_store.cc:276] 0806 20:15:30.363349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.363458 (+ 109us) service_pool.cc:227] Handling call
0806 20:15:30.372139 (+ 8681us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.576800 (+204661us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.576833 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:30.577298 (+ 465us) log.cc:842] Serialized 26 byte log entry
0806 20:15:30.934754 (+357456us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:30.934791 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.664615 (+1729824us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54000901d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:32.664664 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:33.281625 (+616961us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:15:34.356875 (+1075250us) raft_consensus.cc:1671] finished
0806 20:15:34.356913 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:34.357080 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5582,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":4000,"spinlock_wait_cycles":468550144,"thread_start_us":342490,"threads_started":1,"wal-append.queue_time_us":1253999}
I20260806 20:15:34.365614 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.366390 30706 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:34.364866 32122 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.367656 32141 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:34.368321 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 54) took 3624 ms. Trace:
I20260806 20:15:34.368902 32141 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.371687 32047 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.311820 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.323786 32120 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.371404 32128 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:33.275207 32130 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.327633 32060 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:33.775703 32132 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.332569 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1318) took 1155 ms. Trace:
I20260806 20:15:34.334093 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1317) took 1156 ms. Trace:
I20260806 20:15:34.336114 32119 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.336469 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.333389 32067 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:34.333194 32078 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:34.343456 30869 rpcz_store.cc:269] 0806 20:15:30.918292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.919983 (+ 1691us) service_pool.cc:227] Handling call
0806 20:15:33.646281 (+2726298us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54002ba7d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:33.646520 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4198610176}
I20260806 20:15:34.345336 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1168) took 3423 ms. Trace:
I20260806 20:15:34.314488 31421 rpcz_store.cc:276] 0806 20:15:33.144940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.776448 (+631508us) service_pool.cc:227] Handling call
0806 20:15:33.866417 (+ 89969us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:33.866918 (+ 501us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:33.866949 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:33.867010 (+ 61us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:33.867037 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:33.867116 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:34.255286 (+388170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:34.222222 31957 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:34.346939 32079 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:34.348680 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.382808 30871 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:34.351758 32138 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.352020 31418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 260) took 1119 ms. Trace:
W20260806 20:15:34.347493 30788 rpcz_store.cc:269] 0806 20:15:33.018899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.151370 (+132471us) service_pool.cc:227] Handling call
0806 20:15:33.503953 (+352583us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.343413 (+839460us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10410368}
I20260806 20:15:34.348249 30603 catalog_manager.cc:5495] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:34.354064 32049 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.388475 32079 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.389124 32078 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.308869 32033 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.355567 30786 rpcz_store.cc:276] 0806 20:15:33.193647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.670106 (+476459us) service_pool.cc:227] Handling call
0806 20:15:34.310506 (+640400us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.354998 (+ 44492us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14783744}
I20260806 20:15:34.367660 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:33.772105 30868 rpcz_store.cc:276] 0806 20:15:30.134330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.357920 (+223590us) service_pool.cc:227] Handling call
0806 20:15:30.358225 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.742858 (+384633us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.742891 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:30.745438 (+ 2547us) log.cc:842] Serialized 26 byte log entry
0806 20:15:32.179798 (+1434360us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:32.179835 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:32.179897 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:33.213603 (+1033706us) raft_consensus.cc:1671] finished
0806 20:15:33.213628 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:33.213766 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":49479,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":7517,"thread_start_us":1792114,"threads_started":2,"wal-append.queue_time_us":1498143,"wal-append.run_cpu_time_us":19408,"wal-append.run_wall_time_us":1961780}
W20260806 20:15:34.353569 32089 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.600s user 0.008s sys 0.005s
I20260806 20:15:34.378206 30787 rpcz_store.cc:276] 0806 20:15:33.177300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.613209 (+435909us) service_pool.cc:227] Handling call
0806 20:15:34.334036 (+720827us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:34.363754 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:34.359218 32074 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:34.377820 31917 consensus_meta.cc:287] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.729s user 0.009s sys 0.005s
I20260806 20:15:34.379694 31415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 259) took 1147 ms. Trace:
I20260806 20:15:34.353217 30783 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:34.378139 30789 rpcz_store.cc:276] 0806 20:15:33.177590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.714007 (+536417us) service_pool.cc:227] Handling call
0806 20:15:34.332503 (+618496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:34.373136 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.383147 30867 rpcz_store.cc:276] 0806 20:15:30.920981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.520750 (+2599769us) service_pool.cc:227] Handling call
0806 20:15:33.644249 (+123499us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:34.330707 (+686458us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.342995 (+ 12288us) spinlock_profiling.cc:243] Waited 83.7 ms on lock 0x7b54002a0654. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:34.344181 (+ 1186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":192583936}
I20260806 20:15:34.353343 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 56) took 1201 ms. Trace:
I20260806 20:15:34.382754 32067 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:34.331979 32031 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:34.359278 32127 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.370360 30711 rpcz_store.cc:276] 0806 20:15:30.728362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.728533 (+ 171us) service_pool.cc:227] Handling call
0806 20:15:30.728885 (+ 352us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:30.729239 (+ 354us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:30.729266 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:30.729706 (+ 440us) log.cc:842] Serialized 26 byte log entry
0806 20:15:32.002563 (+1272857us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:32.363476 (+360913us) log.cc:842] Serialized 14 byte log entry
0806 20:15:34.333812 (+1970336us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.340529 (+ 6717us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:34.346372 (+ 5843us) raft_consensus.cc:1671] finished
0806 20:15:34.346390 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:34.352920 (+ 6530us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6615,"prepare.queue_time_us":830026,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":654051,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":8818,"thread_start_us":3508291,"threads_started":3,"wal-append.queue_time_us":1317665,"wal-append.run_cpu_time_us":13175,"wal-append.run_wall_time_us":1760706}
I20260806 20:15:34.394575 32119 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.399073 30863 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:34.407635 32067 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:34.403644 31957 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.411571 31957 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.413062 31422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48918 (request call id 257) took 1189 ms. Trace:
I20260806 20:15:34.413870 31422 rpcz_store.cc:276] 0806 20:15:33.223094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.223276 (+ 182us) service_pool.cc:227] Handling call
0806 20:15:33.806660 (+583384us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:34.361423 (+554763us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:34.361472 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:34.361532 (+ 60us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:34.361562 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.412595 (+ 51033us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b54004c2cd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:34.412650 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:34.412824 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1246786432}
I20260806 20:15:34.417766 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.393435 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:34.406126 30703 rpcz_store.cc:276] 0806 20:15:33.146377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.923858 (+777481us) service_pool.cc:227] Handling call
0806 20:15:34.095343 (+171485us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:34.338476 (+243133us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.347473 (+ 8997us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:34.386708 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:34.350709 30861 rpcz_store.cc:269] 0806 20:15:30.917180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.925085 (+ 7905us) service_pool.cc:227] Handling call
0806 20:15:32.753574 (+1828489us) spinlock_profiling.cc:243] Waited 7.31 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71bb7c 00007f714e71b73b 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.331789 (+1578215us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b54000852d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:34.333257 (+ 1468us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1411687680}
I20260806 20:15:34.394212 32079 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.394352 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.420905 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:34.421520 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1320) took 1243 ms. Trace:
I20260806 20:15:34.346024 32150 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.421895 30788 rpcz_store.cc:276] 0806 20:15:33.178077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.394009 (+1215932us) service_pool.cc:227] Handling call
0806 20:15:34.421416 (+ 27407us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:34.422978 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.423877 32150 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.424372 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.425482 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:34.426098 30869 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:34.391991 32049 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.386970 31418 rpcz_store.cc:276] 0806 20:15:33.232397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.233846 (+ 1449us) service_pool.cc:227] Handling call
0806 20:15:33.886950 (+653104us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.351938 (+464988us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":909369088}
I20260806 20:15:34.397419 31419 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:34.389590 32130 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.396338 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 955) took 2392 ms. Trace:
I20260806 20:15:34.374202 31420 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 258) took 1145 ms. Trace:
I20260806 20:15:34.407949 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.377866 32061 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.404047 32120 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.417609 30866 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:34.393935 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.373942 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:34.421801 32129 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:34.399838 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 563712us
I20260806 20:15:34.425225 30788 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:34.347370 32148 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.403339 31415 rpcz_store.cc:276] 0806 20:15:33.232070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.233754 (+ 1684us) service_pool.cc:227] Handling call
0806 20:15:33.877084 (+643330us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71bbce 000000000034aea9 00007f715653ba30 00007f71566017b4 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.379626 (+502542us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24388352}
I20260806 20:15:34.428980 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.429920 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.431684 32031 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.432729 32067 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.436112 32149 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.390966 30870 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:34.438601 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:34.439522 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.402765 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1327) took 1203 ms. Trace:
W20260806 20:15:34.398891 31414 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 255) took 1179 ms (client timeout 1507 ms). Trace:
I20260806 20:15:34.443616 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:34.401665 31917 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.443342 32135 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.444255 32103 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:34.444869 32158 log.cc:924] Time spent T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260806 20:15:34.445996 32010 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:34.446019 30866 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1164) took 3528 ms (client timeout 2034 ms). Trace:
I20260806 20:15:34.449438 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:34.456604 32141 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.457603 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.438079 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1326) took 1238 ms. Trace:
I20260806 20:15:34.460553 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.460753 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.462616 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.468859 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:34.469352 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.470418 31994 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:34.471153 30754 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:34.474751 32046 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:34.477809 30710 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:34.440448 30707 rpcz_store.cc:276] 0806 20:15:31.998544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:32.734093 (+735549us) service_pool.cc:227] Handling call
0806 20:15:32.734393 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:32.734592 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:34.268832 (+1534240us) log.cc:842] Serialized 14 byte log entry
0806 20:15:34.326209 (+ 57377us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:34.337881 (+ 11672us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:34.349461 (+ 11580us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.377040 (+ 27579us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:34.390690 (+ 13650us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5912,"prepare.queue_time_us":98139,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":5619,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":8091,"thread_start_us":1573620,"threads_started":2,"wal-append.queue_time_us":69277}
I20260806 20:15:34.482321 30868 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:34.441383 31420 rpcz_store.cc:276] 0806 20:15:33.228463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.230364 (+ 1901us) service_pool.cc:227] Handling call
0806 20:15:34.029559 (+799195us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71bbce 000000000034aea9 00007f715653ba30 00007f71566017b4 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.374097 (+344538us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":944677120}
I20260806 20:15:34.488291 31421 raft_consensus.cc:2362] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3. Re-sending same reply.
I20260806 20:15:34.492292 30787 rpcz_store.cc:276] 0806 20:15:33.189399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.398048 (+1208649us) service_pool.cc:227] Handling call
0806 20:15:34.427405 (+ 29357us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:34.491407 30866 rpcz_store.cc:269] 0806 20:15:30.918052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:32.781753 (+1863701us) service_pool.cc:227] Handling call
0806 20:15:33.875185 (+1093432us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.445854 (+570669us) spinlock_profiling.cc:243] Waited 71.5 ms on lock 0x7b540014c8d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:34.445954 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":170227328}
W20260806 20:15:34.491894 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 141202us
I20260806 20:15:34.492226 30708 raft_consensus.cc:2354] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:34.478482 30703 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
W20260806 20:15:34.500312 30710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1261) took 1505 ms (client timeout 1659 ms). Trace:
I20260806 20:15:34.501758 30712 raft_consensus.cc:2379] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
W20260806 20:15:34.496351 32155 log.cc:924] Time spent T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Append to log took a long time: real 0.153s user 0.001s sys 0.001s
I20260806 20:15:34.508426 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:34.507992 30710 rpcz_store.cc:269] 0806 20:15:32.993089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.495952 (+502863us) service_pool.cc:227] Handling call
0806 20:15:34.498270 (+1002318us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:34.510345 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 176675us
I20260806 20:15:34.513156 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:34.513733 30706 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:34.501266 30870 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:34.514487 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:34.499264 31418 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 1.
I20260806 20:15:34.524065 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.467273 30791 rpcz_store.cc:276] 0806 20:15:33.193284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.167482 (+974198us) service_pool.cc:227] Handling call
0806 20:15:34.321115 (+153633us) spinlock_profiling.cc:243] Waited 7.92 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.396752 (+ 75637us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18209152}
I20260806 20:15:34.526505 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.528544 30711 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:34.526266 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1323) took 1347 ms. Trace:
I20260806 20:15:34.529928 30789 rpcz_store.cc:276] 0806 20:15:33.178821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.500384 (+1321563us) service_pool.cc:227] Handling call
0806 20:15:34.526200 (+ 25816us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:34.532568 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.533493 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:34.509513 32128 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:34.478257 31414 rpcz_store.cc:269] 0806 20:15:33.219429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.225283 (+ 5854us) service_pool.cc:227] Handling call
0806 20:15:34.172496 (+947213us) spinlock_profiling.cc:243] Waited 7.2 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:34.398827 (+226331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16558848}
I20260806 20:15:34.521776 32047 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:34.542961 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.540699 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1321) took 1362 ms. Trace:
I20260806 20:15:34.543553 30787 rpcz_store.cc:276] 0806 20:15:33.178331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.510809 (+1332478us) service_pool.cc:227] Handling call
0806 20:15:34.540636 (+ 29827us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:34.534610 32026 consensus_meta.cc:287] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.762s user 0.007s sys 0.007s
I20260806 20:15:34.523113 30786 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:34.549636 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.549839 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.553797 30707 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:34.554551 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.554762 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.555472 30707 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:34.555567 32142 tablet_copy_client.cc:316] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.194:42029
I20260806 20:15:34.555943 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.556613 32019 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:34.557557 32019 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.559389 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.560307 30707 raft_consensus.cc:2379] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:34.560384 32019 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.565842 30868 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:34.535565 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.573622 30706 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:34.537570 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.531864 32046 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.507921 32103 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.582926 31917 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.585912 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1324) took 1406 ms. Trace:
I20260806 20:15:34.587352 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:34.568248 32160 log.cc:924] Time spent T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20260806 20:15:34.594568 32128 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:34.595018 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:34.596127 32163 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260806 20:15:34.628165 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.516211 31994 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.596666 32067 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.597076 30673 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:34.541824 32047 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:34.591022 30676 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:34.598834 30835 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:34.563536 32026 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.607704 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.603919 30791 rpcz_store.cc:276] 0806 20:15:33.179053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.538075 (+1359022us) service_pool.cc:227] Handling call
0806 20:15:34.585222 (+ 47147us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:34.585499 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:34.585528 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:34.585570 (+ 42us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:34.585609 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.585679 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:34.585813 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:34.613143 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.616302 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:34.648787 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 51769us
I20260806 20:15:34.656250 30673 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:34.657341 32130 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:34.658111 32130 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.658667 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:34.659333 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 60597us
I20260806 20:15:34.660331 30835 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:34.661424 31957 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:34.617813 30711 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:34.621737 32120 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:34.622115 31388 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:15:34.625254 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 191199us
I20260806 20:15:34.666224 30788 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:34.579272 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:34.633381 32128 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.636179 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 55) took 1490 ms. Trace:
I20260806 20:15:34.636315 32132 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.639675 32047 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:34.643851 32046 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.668789 32150 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:34.675496 32061 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.611989 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.624019 32030 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.684118 32132 consensus_queue.cc:1046] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:34.684504 32030 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:34.645788 30787 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:34.685909 30832 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:34.686251 30792 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:34.677893 31917 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:34.687093 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.687394 31917 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:34.678315 30785 raft_consensus.cc:2379] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:34.685853 31829 raft_consensus.cc:2932] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
W20260806 20:15:34.675757 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 53722.9us
I20260806 20:15:34.689107 32103 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.626127 32089 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:34.667385 30835 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:34.662004 31957 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.663100 32049 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:34.687799 31917 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:34.701568 32089 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.678174 32079 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:34.685945 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1329) took 1492 ms. Trace:
I20260806 20:15:34.687827 31419 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:34.691731 31994 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:34.668933 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 78104.2us
I20260806 20:15:34.670329 30713 rpcz_store.cc:276] 0806 20:15:33.145981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.725692 (+579711us) service_pool.cc:227] Handling call
0806 20:15:33.726316 (+ 624us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:33.726637 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:33.726666 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:33.727327 (+ 661us) log.cc:842] Serialized 26 byte log entry
0806 20:15:34.494228 (+766901us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:34.494252 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.494298 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:34.635959 (+141661us) raft_consensus.cc:1671] finished
0806 20:15:34.635985 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:34.636116 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":234686,"threads_started":1,"wal-append.queue_time_us":811218}
I20260806 20:15:34.668839 32031 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:34.700913 30756 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:34.699225 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:34.687762 32128 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:34.706691 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 43594.2us
I20260806 20:15:34.706104 30787 rpcz_store.cc:276] 0806 20:15:33.193907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.563218 (+1369311us) service_pool.cc:227] Handling call
0806 20:15:34.632327 (+ 69109us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:34.685652 (+ 53325us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.685867 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:34.709599 30866 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:34.713968 31957 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:34.715003 32165 log.cc:924] Time spent T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 20:15:34.812487 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.747982 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:34.750365 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:34.718024 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:34.747484 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 61809.6us
I20260806 20:15:34.717418 30756 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:34.707634 32061 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:34.750555 32031 raft_consensus.cc:686] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:34.712744 30786 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:34.785907 31994 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:34.703116 32135 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.705893 32079 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:34.942909 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1316) took 1765 ms. Trace:
W20260806 20:15:34.941907 30792 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1319) took 1764 ms (client timeout 1646 ms). Trace:
I20260806 20:15:34.707739 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.709558 32060 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:34.870316 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 159210us
I20260806 20:15:34.947471 30754 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:34.879513 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:34.732852 32148 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.957522 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.788544 32130 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:34.825376 32089 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:34.880025 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:34.878892 30784 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:34.770870 30783 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:34.961550 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.768606 32127 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:34.967291 30792 rpcz_store.cc:269] 0806 20:15:33.177844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.517271 (+339427us) service_pool.cc:227] Handling call
0806 20:15:34.941678 (+1424407us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54003b1fd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:34.941827 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2422180608}
I20260806 20:15:34.833735 32047 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:34.960843 31957 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:34.975497 32101 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:34.967150 30789 rpcz_store.cc:276] 0806 20:15:33.176999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.530549 (+1353550us) service_pool.cc:227] Handling call
0806 20:15:34.599301 (+ 68752us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:34.648243 (+ 48942us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:34.655971 (+ 7728us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:34.665113 (+ 9142us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:34.665150 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.942669 (+277519us) spinlock_profiling.cc:243] Waited 39.6 ms on lock 0x7b54001d1ad4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:34.942716 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:34.942850 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91146112}
W20260806 20:15:34.981176 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 263829us
I20260806 20:15:34.977423 32010 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:34.967991 32079 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:34.965924 32120 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:34.985838 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 274185us
I20260806 20:15:34.959342 31994 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:34.970366 32060 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:34.956012 31917 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:34.974576 32103 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:34.994212 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:34.973429 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 37) took 1502 ms. Trace:
W20260806 20:15:34.990440 32027 consensus_meta.cc:287] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.608s user 0.010s sys 0.002s
W20260806 20:15:34.989895 30783 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1325) took 1810 ms (client timeout 1936 ms). Trace:
I20260806 20:15:35.001637 30835 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.002431 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.005739 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:34.989650 30784 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1331) took 1468 ms (client timeout 1659 ms). Trace:
I20260806 20:15:34.995298 31994 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:34.989578 32101 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:34.997897 32060 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:35.015430 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:35.014112 30792 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:34.983114 32135 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260806 20:15:34.995826 31416 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:35.006537 31957 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:35.020159 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1330) took 1495 ms. Trace:
I20260806 20:15:35.020709 30787 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:35.003538 30864 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:35.007911 32027 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.075969 30790 raft_consensus.cc:2379] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:35.076702 32027 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.051422 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:34.995035 32127 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.026578 30788 rpcz_store.cc:276] 0806 20:15:33.520642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.995708 (+1475066us) service_pool.cc:227] Handling call
0806 20:15:35.008223 (+ 12515us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:35.015787 (+ 7564us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:35.015815 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:35.015933 (+ 118us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:35.015968 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:35.016029 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:35.016139 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:35.029490 30676 leader_election.cc:398] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3.
I20260806 20:15:35.078541 30676 leader_election.cc:401] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 20:15:35.018424 32101 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:35.074460 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 81342.6us
I20260806 20:15:35.003374 32120 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.041805 32013 raft_consensus.cc:3032] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 20:15:35.003592 30783 rpcz_store.cc:269] 0806 20:15:33.179435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.864127 (+684692us) service_pool.cc:227] Handling call
0806 20:15:34.988513 (+1124386us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:35.091544 30835 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.000288 30786 rpcz_store.cc:276] 0806 20:15:33.471501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.605211 (+1133710us) service_pool.cc:227] Handling call
0806 20:15:34.671514 (+ 66303us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:34.957058 (+285544us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:34.971749 (+ 14691us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:35.082753 31917 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:35.011425 30784 rpcz_store.cc:269] 0806 20:15:33.521121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:34.358639 (+837518us) service_pool.cc:227] Handling call
0806 20:15:34.988409 (+629770us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:35.084975 30754 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:35.132351 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 52334.6us
I20260806 20:15:35.132648 30756 leader_election.cc:398] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:35.133066 30756 leader_election.cc:401] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260806 20:15:35.134109 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 184048us
I20260806 20:15:35.092619 32013 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:35.104372 32105 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:35.080150 32031 consensus_queue.cc:237] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.136669 31829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:53728 (request call id 6) took 5358 ms. Trace:
I20260806 20:15:35.136750 32196 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:35.137394 31957 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:15:35.136343 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 57768us
W20260806 20:15:35.143113 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 129260us
I20260806 20:15:35.143620 31389 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.129776 32138 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3.
I20260806 20:15:35.146525 30756 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:35.149792 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 63269.1us
I20260806 20:15:35.142127 32019 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.144411 32105 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:35.153040 32019 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:35.154573 32019 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.154706 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.156167 30835 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:35.156558 32187 log.cc:924] Time spent T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260806 20:15:35.157224 32199 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.157704 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:35.158254 31957 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.158468 31917 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.159188 31917 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:35.159598 31917 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.159933 30676 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:35.160862 32061 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:15:35.161836 32138 consensus_queue.cc:1035] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:35.162331 30835 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.163717 32138 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.164098 32138 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:35.164304 32089 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:35.164449 32138 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.164739 32089 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:35.165145 32089 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:35.158679 31957 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.165865 32105 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:35.165970 31957 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.171205 32120 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.145356 31829 rpcz_store.cc:276] 0806 20:15:29.778445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:29.778977 (+ 532us) service_pool.cc:227] Handling call
0806 20:15:30.055806 (+276829us) log.cc:842] Serialized 513 byte log entry
0806 20:15:30.291029 (+235223us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b18002f3820. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516abf11 00007f7151600f91 00007f71515e9545 00007f71515ee771 00007f71515ee6f9 00007f71515ee689 00007f71515ee651 00007f71515ed81d 00007f71579ab204 00007f71579ab039
0806 20:15:35.136511 (+4845482us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3026,"raft.queue_time_us":11275,"raft.run_cpu_time_us":15961,"raft.run_wall_time_us":152497,"spinlock_wait_cycles":5255424,"thread_start_us":95114,"threads_started":1,"wal-append.queue_time_us":117907,"wal-append.run_cpu_time_us":1721,"wal-append.run_wall_time_us":1001738}
I20260806 20:15:35.174441 32127 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.176785 32026 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:35.184876 30835 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.148921 32103 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:35.186452 32007 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.186601 32130 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 20:15:35.184952 30790 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1154) took 2042 ms (client timeout 2047 ms). Trace:
I20260806 20:15:35.150408 32047 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:35.151759 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.154059 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:35.191376 30790 rpcz_store.cc:269] 0806 20:15:33.142219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.142329 (+ 110us) service_pool.cc:227] Handling call
0806 20:15:33.506233 (+363904us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:35.178875 (+1672642us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54002832d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:35.184881 (+ 6006us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3262851712}
I20260806 20:15:35.196193 31419 pending_rounds.cc:76] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Aborting all ops after (but not including) 4
I20260806 20:15:35.202574 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:35.203112 32192 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260806 20:15:35.204622 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.205515 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.205853 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.207129 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.208738 32089 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.209748 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.189692 32007 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:35.211961 31419 pending_rounds.cc:97] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260806 20:15:35.237221 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.262418 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:35.224897 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.278041 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.276556 30787 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:35.232110 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.169123 32196 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:35.280066 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 136492us
W20260806 20:15:35.234014 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 54753.8us
W20260806 20:15:35.272822 32189 log.cc:924] Time spent T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20260806 20:15:35.281980 32019 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.287940 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.283025 31994 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.284508 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.285326 31389 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:35.285732 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.280853 31419 raft_consensus.cc:2866] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:35.289840 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:35.289238 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.265286 32079 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.263039 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:35.235857 32007 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.233592 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.219671 32130 raft_consensus.cc:686] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:35.293435 30869 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:35.286371 30861 raft_consensus.cc:2354] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:35.295462 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.296595 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:35.252600 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.231458 30783 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:35.297964 31389 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.234340 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.299408 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.311406 30789 raft_consensus.cc:1226] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:35.296013 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.224434 31957 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.283880 32089 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:35.302249 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.291107 32105 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:35.304359 32178 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.126s user 0.001s sys 0.000s
I20260806 20:15:35.310665 30866 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:35.299175 30864 raft_consensus.cc:2379] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:35.314543 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.323531 30792 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:35.311249 30710 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
W20260806 20:15:35.314079 30785 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1322) took 2133 ms (client timeout 1966 ms). Trace:
I20260806 20:15:35.316644 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.317003 30786 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:35.299434 32195 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.318343 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.318588 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.318883 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.317344 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.325281 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.325505 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.310995 32019 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:35.325085 30785 rpcz_store.cc:269] 0806 20:15:33.178572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:33.864401 (+685829us) service_pool.cc:227] Handling call
0806 20:15:35.301273 (+1436872us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b54000131d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:35.311712 (+ 10439us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":666884480}
I20260806 20:15:35.329694 32196 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.331943 30756 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.332794 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:35.328769 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.338246 30835 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.338583 32130 consensus_queue.cc:237] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.339486 30757 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:35.340831 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.361665 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.341953 30870 raft_consensus.cc:1226] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:15:35.342550 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.343279 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.365845 30868 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:35.343912 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.344619 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:35.345068 30712 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:35.325847 32127 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.211162 32013 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.366961 32127 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:35.367383 32127 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.367527 32013 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:35.353173 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.357156 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.352936 32089 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.357936 30834 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.368849 30787 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:35.333915 32195 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.375449 30790 raft_consensus.cc:1261] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:35.376423 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.378664 31829 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.378981 30703 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:35.381726 32148 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.382125 32148 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:35.382452 32148 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.385697 32089 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.387887 32089 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:35.333242 32101 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.318599 31917 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:35.354372 32203 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260806 20:15:35.377017 32195 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.402042 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.361914 31416 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.396734 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.394850 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:35.398473 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 59073.2us
I20260806 20:15:35.397414 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.395325 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.356762 32061 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:35.410408 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.411059 30867 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:35.411921 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.398718 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.413029 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:35.400272 32204 log.cc:924] Time spent T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.106s user 0.002s sys 0.000s
I20260806 20:15:35.413823 30757 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.414101 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.415166 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:35.415359 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.415771 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.416616 30863 raft_consensus.cc:2379] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:35.416863 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.417493 30791 raft_consensus.cc:2362] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3. Re-sending same reply.
I20260806 20:15:35.417811 30863 raft_consensus.cc:1261] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:35.418304 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:35.419004 31415 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.416625 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.395889 32010 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:35.418741 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.424979 32010 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:35.426080 32138 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.428254 32138 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.421653 32027 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:35.400703 32141 raft_consensus.cc:979] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207: : Instructing follower 42cabca48db7403b8523a3382a046043 to start an election
I20260806 20:15:35.400503 31994 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.396420 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.397934 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.429745 30711 raft_consensus.cc:2379] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:35.410082 32060 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.431334 30867 raft_consensus.cc:2379] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:35.431932 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:35.432973 32060 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:35.408954 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.371358 32019 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.396106 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.405413 30835 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:35.438807 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 47008.3us
I20260806 20:15:35.439556 30835 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.440821 30835 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.441821 32061 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.442440 32061 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:35.442998 32061 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:35.421437 30757 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:35.444519 30757 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.445847 30757 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:35.391407 30712 raft_consensus.cc:1261] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:35.422828 30713 raft_consensus.cc:2379] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:35.409982 31917 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.446511 32122 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.461243 32122 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.461731 32122 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.437266 32027 raft_consensus.cc:686] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:35.463142 32027 consensus_queue.cc:237] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.447009 32047 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.447238 32141 raft_consensus.cc:1067] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Signalling peer 42cabca48db7403b8523a3382a046043 to start an election
I20260806 20:15:35.451131 30706 consensus_queue.cc:237] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:35.379056 31829 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.401350 31957 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:35.415390 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1336) took 1074 ms. Trace:
I20260806 20:15:35.470043 30789 rpcz_store.cc:276] 0806 20:15:34.341259 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:35.010933 (+669674us) service_pool.cc:227] Handling call
0806 20:15:35.022228 (+ 11295us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:35.314252 (+292024us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:35.415179 (+100927us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54003a07d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:35.415336 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":650414848}
I20260806 20:15:35.469229 31829 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.471385 32010 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.474231 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.474787 30713 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:35.475760 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.476526 30708 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:35.477643 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.478626 30870 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:35.479012 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.479125 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.482555 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.452499 32007 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.456311 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:35.484508 31422 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.484120 30787 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
W20260806 20:15:35.456766 32205 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260806 20:15:35.458124 32105 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:35.371045 32079 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:35.458580 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:35.459650 30835 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.485499 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.487419 30866 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:35.378058 30834 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:35.493198 32046 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:35.494143 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 48524.6us
I20260806 20:15:35.497982 30757 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.404246 32101 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:35.494999 30712 raft_consensus.cc:2354] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:35.406898 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.487888 30869 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:35.502341 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:35.503273 30870 raft_consensus.cc:2379] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:35.504004 30789 raft_consensus.cc:1226] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:35.506507 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:35.507681 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 129675us
I20260806 20:15:35.508251 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:15:35.510001 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 50438.6us
I20260806 20:15:35.510574 30835 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.511731 32010 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:35.513019 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:35.514559 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:35.519495 30835 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.520533 30835 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.423568 32127 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:35.507840 30676 connection.cc:232] RPC call timeout handler was delayed by -3.5825707979965955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
I20260806 20:15:35.500108 32046 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.529558 32061 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.532161 30713 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:35.554029 30710 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:35.533596 30703 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:35.536437 32007 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.516709 32078 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:35.499631 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:35.563361 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.497699 31957 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:35.553215 32061 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.541599 30712 raft_consensus.cc:1226] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:35.543290 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:35.543841 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:35.544458 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:35.549392 30832 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.483639 32148 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.552794 30789 raft_consensus.cc:1226] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:35.589870 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.590022 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.591200 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.591835 32141 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:35.592072 30703 raft_consensus.cc:2379] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:35.554948 31388 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:35.566488 32130 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:35.557885 31422 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:35.557776 31389 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:35.519237 32060 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.538811 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:35.599313 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 50004.2us
I20260806 20:15:35.562446 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:35.562572 30804 tablet_copy_service.cc:140] P d4181d4d907f477f9feddc44ba3b9207: Received BeginTabletCopySession request for tablet 5ba1969a3911423e8e51fc4f2962515d from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:53732)
I20260806 20:15:35.521530 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:35.562878 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.564286 30713 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260806 20:15:35.601563 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 46724.2us
I20260806 20:15:35.536489 30757 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:35.570473 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:35.563601 31417 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:35.563941 30784 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 20:15:35.597713 32060 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:35.609555 32060 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.609935 32089 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.610651 30804 tablet_copy_service.cc:161] P d4181d4d907f477f9feddc44ba3b9207: Beginning new tablet copy session on tablet 5ba1969a3911423e8e51fc4f2962515d from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:53732: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-5ba1969a3911423e8e51fc4f2962515d
I20260806 20:15:35.614688 32119 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.615120 32119 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.576001 32078 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:35.554515 32046 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:35.578145 30785 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:35.579420 32195 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.582109 30791 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:15:35.583945 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.587085 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.591274 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.619196 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.555832 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:35.554905 30676 connection.cc:232] RPC call timeout handler was delayed by -2.813914655969711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260806 20:15:35.620021 30711 raft_consensus.cc:2354] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:35.584374 32061 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:35.618119 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 60440.1us
I20260806 20:15:35.605216 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.606433 30603 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41560 (request call id 216) took 5369 ms. Trace:
I20260806 20:15:35.606588 30786 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:35.607272 32013 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.604044 32135 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 20:15:35.607080 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 70698.5us
I20260806 20:15:35.606592 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.627592 30711 raft_consensus.cc:1226] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:35.607954 32130 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.615481 32119 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:35.617211 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 96727.3us
W20260806 20:15:35.619941 30676 connection.cc:232] RPC call timeout handler was delayed by -2.6870146559667774s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
I20260806 20:15:35.597832 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.607065 32122 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.624944 31389 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:35.626631 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.630441 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.631249 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:35.632906 30676 connection.cc:232] RPC call timeout handler was delayed by -2.6474146559776273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20260806 20:15:35.628525 30707 raft_consensus.cc:2354] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:35.632598 30835 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.596524 31829 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.637521 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:35.640030 30676 connection.cc:232] RPC call timeout handler was delayed by -2.4584146559936926s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20260806 20:15:35.617794 32078 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:35.643344 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.626503 32013 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.647383 32010 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.647871 32010 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.648221 32010 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.641906 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:35.645622 30676 connection.cc:232] RPC call timeout handler was delayed by -2.4143146559945308s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2034s for a response
I20260806 20:15:35.640362 32130 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:35.662034 30787 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:35.652750 30676 connection.cc:232] RPC call timeout handler was delayed by -2.4026146559917834s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20470000000000002s for a response
W20260806 20:15:35.666818 30676 connection.cc:232] RPC call timeout handler was delayed by -2.3882146559772082s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
W20260806 20:15:35.667095 30676 connection.cc:232] RPC call timeout handler was delayed by -2.371114655979909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
W20260806 20:15:35.667307 30676 connection.cc:232] RPC call timeout handler was delayed by -2.0292361799802165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30810000000000004s for a response
I20260806 20:15:35.654608 31389 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:35.667949 30832 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.669618 30832 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:35.670909 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.671628 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.672432 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:35.673154 31420 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:35.673840 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.649994 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.675415 30783 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:35.650992 32078 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:35.677958 30756 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.678259 32127 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.659300 32130 consensus_queue.cc:1035] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260806 20:15:35.679312 30756 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.677948 32195 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:35.680691 32103 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:35.681107 32103 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:35.681632 32105 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.682932 30756 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:35.683751 32103 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:35.621713 30603 rpcz_store.cc:276] 0806 20:15:30.236957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.240718 (+ 3761us) service_pool.cc:227] Handling call
0806 20:15:35.606293 (+5365575us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:15:34.607513 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:15:34.608136 (+ 623us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:15:34.608169 (+ 33us) write_op.cc:435] Acquired schema lock
0806 20:15:34.608181 (+ 12us) tablet.cc:660] Decoding operations
0806 20:15:34.608868 (+ 687us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:15:34.608931 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0806 20:15:34.608967 (+ 36us) tablet.cc:683] Acquiring locks for 3 operations
0806 20:15:34.609304 (+ 337us) tablet.cc:699] Row locks acquired
0806 20:15:34.609319 (+ 15us) write_op.cc:260] PREPARE: finished
0806 20:15:34.609496 (+ 177us) write_op.cc:270] Start()
0806 20:15:34.609623 (+ 127us) write_op.cc:276] Timestamp: P: 1786047334609428 usec, L: 0
0806 20:15:34.609646 (+ 23us) op_driver.cc:331] REPLICATION: starting
0806 20:15:35.234958 (+625312us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b54001c19a0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f7151604dc0 00007f7151675bd2 00007f715166b2d7 00007f715166b114 00007f715507eade 00007f715507e2b8 00007f7155082ba1 00007f7155082b59 00007f7155082ae9 00007f7155082ab1
0806 20:15:35.296943 (+ 61985us) log.cc:842] Serialized 1321 byte log entry
0806 20:15:35.426851 (+129908us) op_driver.cc:444] REPLICATION: finished
0806 20:15:35.514295 (+ 87444us) write_op.cc:301] APPLY: starting
0806 20:15:35.514393 (+ 98us) tablet.cc:1365] starting BulkCheckPresence
0806 20:15:35.514555 (+ 162us) tablet.cc:1368] finished BulkCheckPresence
0806 20:15:35.514591 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:15:35.515347 (+ 756us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:15:35.515621 (+ 274us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0806 20:15:35.515699 (+ 78us) write_op.cc:312] APPLY: finished
0806 20:15:35.516017 (+ 318us) log.cc:842] Serialized 35 byte log entry
0806 20:15:35.566153 (+ 50136us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:35.581604 (+ 15451us) write_op.cc:454] Released schema lock
0806 20:15:35.581688 (+ 84us) write_op.cc:341] FINISH: Updating metrics
0806 20:15:35.583758 (+ 2070us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:35.586005 (+ 2247us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":36503,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":87175,"apply.run_cpu_time_us":3040,"apply.run_wall_time_us":82464,"num_ops":3,"prepare.queue_time_us":180353,"prepare.run_cpu_time_us":29696,"prepare.run_wall_time_us":695880,"replication_time_us":817094,"spinlock_wait_cycles":86734720,"thread_start_us":222507,"threads_started":2}]]}
I20260806 20:15:35.682958 30602 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34006 (request call id 124) took 5062 ms. Trace:
W20260806 20:15:35.691919 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 134856us
I20260806 20:15:35.697896 30834 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.639170 32135 raft_consensus.cc:686] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:35.704007 30789 raft_consensus.cc:1226] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:35.704351 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 49790.6us
I20260806 20:15:35.706669 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.707804 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.649832 32089 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.709736 30756 leader_election.cc:398] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:35.718154 30756 leader_election.cc:401] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 20:15:35.665271 32013 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.719913 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:35.720707 30785 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:35.717219 32089 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.721964 32089 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.723325 31415 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:35.710527 32127 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.685299 30712 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3"
dest_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:59116
I20260806 20:15:35.706403 32141 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.725901 30754 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.726466 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:35.716770 31417 raft_consensus.cc:1226] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:35.726737 32141 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:35.727115 31829 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.727334 32141 raft_consensus.cc:686] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:35.725775 30712 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:15:35.728139 30712 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.725219 32127 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.681120 32130 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:35.664605 32148 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:35.718820 30757 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:35.729336 30834 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.729421 30835 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.733671 30601 catalog_manager.cc:5495] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:35.737500 32135 consensus_queue.cc:237] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.683769 32203 consensus_queue.cc:1046] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260806 20:15:35.703796 31416 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:35.748098 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.747184 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.663920 32060 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:35.759917 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 90383.6us
I20260806 20:15:35.761922 30832 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:35.762473 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 62032.7us
I20260806 20:15:35.731943 31957 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.764701 30756 leader_election.cc:398] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:35.732189 32127 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.696722 30602 rpcz_store.cc:276] 0806 20:15:30.621100 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.621711 (+ 611us) service_pool.cc:227] Handling call
0806 20:15:35.682875 (+5061164us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:15:33.329690 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:15:33.615967 (+286277us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:15:33.616053 (+ 86us) write_op.cc:435] Acquired schema lock
0806 20:15:33.616096 (+ 43us) tablet.cc:660] Decoding operations
0806 20:15:33.617075 (+ 979us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:15:33.617172 (+ 97us) write_op.cc:641] Partition lock acquired for write op
0806 20:15:33.617220 (+ 48us) tablet.cc:683] Acquiring locks for 3 operations
0806 20:15:33.617795 (+ 575us) tablet.cc:699] Row locks acquired
0806 20:15:33.617831 (+ 36us) write_op.cc:260] PREPARE: finished
0806 20:15:33.618101 (+ 270us) write_op.cc:270] Start()
0806 20:15:33.618292 (+ 191us) write_op.cc:276] Timestamp: P: 1786047333617997 usec, L: 0
0806 20:15:33.618332 (+ 40us) op_driver.cc:331] REPLICATION: starting
0806 20:15:33.619959 (+ 1627us) log.cc:842] Serialized 1311 byte log entry
0806 20:15:35.316908 (+1696949us) op_driver.cc:444] REPLICATION: finished
0806 20:15:35.426467 (+109559us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b5400002dd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166c689 00007f715160a804 00007f715160a7b3 00007f715160a728 00007f715160a6e3 00007f71516099cc 00007f7151633a16 00007f715161b5cb 00007f7151605c8d 00007f7151609665
0806 20:15:35.614907 (+188440us) spinlock_profiling.cc:243] Waited 7.35 ms on lock 0x7b180023b5a0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e753638 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d 00007f71579ab204 00007f71579ab039 00007f714e735dc6
0806 20:15:35.615052 (+ 145us) write_op.cc:301] APPLY: starting
0806 20:15:35.615128 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0806 20:15:35.615251 (+ 123us) tablet.cc:1368] finished BulkCheckPresence
0806 20:15:35.615272 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:15:35.618459 (+ 3187us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:15:35.618614 (+ 155us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0806 20:15:35.618661 (+ 47us) write_op.cc:312] APPLY: finished
0806 20:15:35.640227 (+ 21566us) log.cc:842] Serialized 35 byte log entry
0806 20:15:35.658015 (+ 17788us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:35.658131 (+ 116us) write_op.cc:454] Released schema lock
0806 20:15:35.658169 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0806 20:15:35.658444 (+ 275us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:35.658471 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":50019,"apply.run_cpu_time_us":2685,"apply.run_wall_time_us":43684,"num_ops":3,"prepare.queue_time_us":416118,"prepare.run_cpu_time_us":10449,"prepare.run_wall_time_us":400747,"raft.queue_time_us":487497,"raft.run_cpu_time_us":30652,"raft.run_wall_time_us":487336,"replication_time_us":1698469,"spinlock_wait_cycles":647515264,"thread_start_us":1619170,"threads_started":4,"wal-append.queue_time_us":698062}]]}
I20260806 20:15:35.776906 30754 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:35.777560 31418 raft_consensus.cc:1226] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:35.775389 32130 consensus_queue.cc:1046] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260806 20:15:35.778240 32218 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 20:15:35.779304 32216 log.cc:924] Time spent T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20260806 20:15:35.757042 32148 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:35.774878 31957 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:35.757488 32101 consensus_queue.cc:1046] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20260806 20:15:35.757925 31829 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.757658 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:35.792955 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 59873us
I20260806 20:15:35.790082 30712 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.757673 32007 consensus_queue.cc:1046] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20260806 20:15:35.747321 32026 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:35.802894 31957 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.787320 32122 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.808137 30712 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.810217 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.826579 30785 raft_consensus.cc:1226] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:35.827862 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.830269 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:35.831077 30792 raft_consensus.cc:2379] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 2: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:35.837224 32135 consensus_queue.cc:1046] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:35.838702 32046 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:35.841291 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 115544us
I20260806 20:15:35.757565 32141 consensus_queue.cc:237] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:35.845137 31388 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:35.851048 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 89230.6us
I20260806 20:15:35.852303 30832 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.851323 32203 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.853608 30832 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:35.855110 32218 raft_consensus.cc:686] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:35.856688 32218 consensus_queue.cc:237] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.868340 30756 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.869354 30673 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:35.863629 31416 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:35.841471 32105 consensus_queue.cc:1046] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
W20260806 20:15:35.878163 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 109100us
W20260806 20:15:35.886616 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 139495us
I20260806 20:15:35.887405 30834 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.849815 32103 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.849385 32061 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:35.877856 32213 log.cc:924] Time spent T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260806 20:15:35.888495 30834 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.927011 30834 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:35.871738 32013 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.888500 32222 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:35.888484 32233 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:35.917821 32046 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.870322 32019 consensus_queue.cc:1046] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260806 20:15:35.876822 32130 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:35.934297 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 52626.6us
I20260806 20:15:35.877993 32223 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:35.959297 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:35.961171 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 84438.2us
I20260806 20:15:35.963563 32229 consensus_queue.cc:1046] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:35.964907 32013 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:35.965706 32130 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.966123 32130 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:35.966361 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:35.966599 32046 raft_consensus.cc:3032] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:35.966945 32046 raft_consensus.cc:729] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:35.968062 32224 raft_consensus.cc:3032] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:35.967733 32046 consensus_queue.cc:260] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:35.968451 32224 raft_consensus.cc:729] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:35.964937 32229 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:35.969544 32229 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:35.969920 32229 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.889941 31917 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:35.976092 32232 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:35.925938 32060 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:35.889662 32220 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:35.982472 30834 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:35.980332 30673 consensus_peers.cc:487] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa7cfe759ca64ca89a0529c2d3da5bf7. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:35.965292 32030 raft_consensus.cc:3032] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 20:15:36.018420 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 137278us
I20260806 20:15:35.973007 32222 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:15:36.026873 32214 log.cc:924] Time spent T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.307s user 0.003s sys 0.000s
I20260806 20:15:36.030180 32217 raft_consensus.cc:3032] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:36.030714 32217 raft_consensus.cc:729] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:36.031629 32217 consensus_queue.cc:260] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:36.033959 32225 raft_consensus.cc:3032] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:36.034385 32225 raft_consensus.cc:729] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
W20260806 20:15:36.077068 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 55344.6us
I20260806 20:15:35.973817 32130 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:35.980970 32233 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:36.077548 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 95265.8us
I20260806 20:15:36.033231 32217 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:36.017099 32220 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:36.001957 30786 raft_consensus.cc:1226] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:36.095203 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 124397us
I20260806 20:15:36.112051 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.014482 32231 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:35.980227 31829 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:36.107838 30834 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.001421 31917 raft_consensus.cc:686] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:36.127894 31917 consensus_queue.cc:237] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:36.129840 31421 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:36.019377 32223 raft_consensus.cc:686] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:36.088001 32046 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:36.132373 32223 consensus_queue.cc:237] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:36.090529 32195 raft_consensus.cc:3032] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:35.969262 32224 consensus_queue.cc:260] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:36.133607 32195 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:36.134361 32224 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:36.108166 32060 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:36.141880 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.116008 32220 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:36.118345 32231 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:36.120924 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.157786 31389 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:36.097973 32149 consensus_queue.cc:1046] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260806 20:15:36.140575 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.167387 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.018981 32030 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:36.168380 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.168248 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:36.170122 30712 raft_consensus.cc:2362] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3. Re-sending same reply.
I20260806 20:15:36.168231 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:36.148419 32240 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:36.172994 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.176476 30713 raft_consensus.cc:2379] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:36.172780 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:36.176899 31389 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.160197 32231 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:36.199517 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.168555 32218 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:36.143875 32138 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260806 20:15:36.168480 32148 raft_consensus.cc:686] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:36.182607 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.210372 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.172153 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.203740 32061 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:36.253688 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.188326 32103 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:36.202726 32120 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:36.278193 30711 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:36.150758 32222 consensus_queue.cc:1035] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260806 20:15:36.329030 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:36.336519 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:36.338282 30834 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:36.198308 32089 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:36.354265 30834 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.339166 30756 leader_election.cc:398] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:36.373963 32061 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:36.268136 30785 raft_consensus.cc:1261] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:36.381392 30756 leader_election.cc:401] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 20:15:36.383778 32089 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:36.378751 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.385490 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:36.398079 30713 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:36.386656 32237 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:36.324946 31957 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:36.387662 32221 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 20:15:36.389509 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 50596.2us
W20260806 20:15:36.388734 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 206225us
I20260806 20:15:36.378052 30757 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:36.388914 32148 consensus_queue.cc:237] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:36.390563 31421 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:36.316105 30784 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260806 20:15:36.385986 32239 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.130s user 0.001s sys 0.000s
W20260806 20:15:36.403177 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 60467.5us
I20260806 20:15:36.394829 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:36.346764 32078 consensus_meta.cc:287] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.671s user 0.009s sys 0.002s
I20260806 20:15:36.344676 32149 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:15:36.426191 32026 consensus_meta.cc:287] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.615s user 0.008s sys 0.003s
I20260806 20:15:36.455636 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:36.455616 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 77723.7us
I20260806 20:15:36.467692 30757 leader_election.cc:398] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:36.461261 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:36.469708 30757 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:36.467263 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 287889us
W20260806 20:15:36.465065 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 288207us
I20260806 20:15:36.435231 32061 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:36.472338 30835 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.482761 31389 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.426725 32221 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:36.431734 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:36.489246 30835 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.297806 32030 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:36.536000 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 87247.6us
W20260806 20:15:36.541064 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 51988.3us
W20260806 20:15:36.543949 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 67096.9us
I20260806 20:15:36.408231 32234 consensus_queue.cc:1046] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.114s
I20260806 20:15:36.545375 31389 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.496187 32221 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:36.547688 32010 consensus_meta.cc:287] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.899s user 0.014s sys 0.000s
W20260806 20:15:36.564291 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 94795.7us
I20260806 20:15:36.562392 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:36.475664 32026 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:36.476089 30708 raft_consensus.cc:1261] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:36.565201 30676 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:36.574819 30788 raft_consensus.cc:1226] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:36.544598 32061 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 20:15:36.558708 30783 consensus_meta.cc:287] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.878s user 0.006s sys 0.007s
I20260806 20:15:36.461934 32078 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:36.599007 30757 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:36.611553 30711 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:36.548743 32120 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:36.496774 32138 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:36.549115 32254 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:36.559913 32203 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:36.493534 32141 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:36.632288 30871 raft_consensus.cc:2379] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:36.608526 30783 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:36.642684 30869 raft_consensus.cc:1226] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:36.642742 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 43969.6us
I20260806 20:15:36.554517 32149 consensus_queue.cc:1046] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260806 20:15:36.559290 32201 consensus_queue.cc:1046] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
W20260806 20:15:36.652804 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 87568.3us
I20260806 20:15:36.619154 32244 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:36.602038 32074 consensus_queue.cc:1046] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260806 20:15:36.634624 32138 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:36.640372 32236 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:36.657153 32103 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:36.641906 32203 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:36.603351 32010 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:36.690357 30713 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
W20260806 20:15:36.722934 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 182228us
W20260806 20:15:36.728641 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 106083us
I20260806 20:15:36.688778 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1163) took 1097 ms. Trace:
I20260806 20:15:36.630263 32026 consensus_queue.cc:1046] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260806 20:15:36.714202 32138 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:36.707448 30784 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 20:15:36.724665 32203 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:36.760311 30676 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:36.768093 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:36.734297 31957 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:36.735152 32103 raft_consensus.cc:686] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:36.777288 31421 raft_consensus.cc:1226] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:36.779832 32248 log.cc:924] Time spent T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.401s user 0.002s sys 0.000s
I20260806 20:15:36.783468 32242 consensus_queue.cc:1035] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.180s
I20260806 20:15:36.784708 32010 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:36.749603 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:36.785301 32227 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:15:36.786942 32231 consensus_meta.cc:287] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.526s user 0.010s sys 0.009s
I20260806 20:15:36.769320 31417 raft_consensus.cc:1226] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:36.770131 32046 consensus_meta.cc:287] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.638s user 0.014s sys 0.001s
I20260806 20:15:36.785945 32228 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:36.812484 30866 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:36.738492 30783 rpcz_store.cc:276] 0806 20:15:35.575153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:35.624125 (+ 48972us) service_pool.cc:227] Handling call
0806 20:15:36.672895 (+1048770us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:36.786729 32252 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:36.836736 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:36.839001 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 163408us
I20260806 20:15:36.711056 32266 consensus_queue.cc:1046] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
W20260806 20:15:36.850368 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 288726us
I20260806 20:15:36.790683 32236 consensus_queue.cc:1046] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:36.765188 32201 consensus_queue.cc:1046] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260806 20:15:36.865813 31421 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:36.729058 32244 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:36.858183 30713 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:36.855409 30789 raft_consensus.cc:1226] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:36.878772 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.883339 32236 raft_consensus.cc:3032] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:36.884056 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:36.888305 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:36.856489 30708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f"
dest_uuid: "42cabca48db7403b8523a3382a046043"
mode: GRACEFUL
new_leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
from {username='slave'} at 127.0.0.1:38516
I20260806 20:15:36.787109 32256 consensus_queue.cc:1046] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.237s
I20260806 20:15:36.898192 32256 raft_consensus.cc:3032] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:36.898977 32256 raft_consensus.cc:729] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:36.903424 32061 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:36.904435 32127 consensus_meta.cc:287] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.112s user 0.015s sys 0.002s
I20260806 20:15:36.936761 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:36.938133 30866 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 4
I20260806 20:15:36.889046 32270 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:36.940310 32061 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:36.856982 32254 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:36.961465 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 109472us
I20260806 20:15:36.818528 32231 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:36.959806 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:36.892786 32078 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:36.896839 32236 raft_consensus.cc:729] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:36.914520 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:36.905251 32127 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:36.861023 32252 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:36.900804 32256 consensus_queue.cc:260] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:37.136762 32265 log.cc:924] Time spent T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.198s user 0.000s sys 0.001s
W20260806 20:15:37.058857 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 153758us
I20260806 20:15:37.140276 32244 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:37.141381 32258 log.cc:924] Time spent T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.310s user 0.002s sys 0.000s
I20260806 20:15:37.141930 30790 raft_consensus.cc:1261] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:15:36.965581 30757 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:37.145944 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 180510us
W20260806 20:15:37.070019 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 325111us
I20260806 20:15:37.146819 30866 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
W20260806 20:15:36.936759 32060 consensus_meta.cc:287] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.789s user 0.011s sys 0.008s
W20260806 20:15:37.020725 32253 log.cc:924] Time spent T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.389s user 0.000s sys 0.002s
W20260806 20:15:37.028122 32272 log.cc:924] Time spent T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20260806 20:15:37.136583 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:37.179186 30757 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:37.181663 30834 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:37.184963 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:36.951365 32270 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:36.963112 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:36.984045 32254 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:36.978031 32026 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.221s
I20260806 20:15:37.102667 32277 raft_consensus.cc:3032] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
W20260806 20:15:37.233098 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 67437.4us
W20260806 20:15:37.177700 32119 consensus_meta.cc:287] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.546s user 0.015s sys 0.009s
I20260806 20:15:37.027907 31416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 246) took 1196 ms. Trace:
I20260806 20:15:37.177536 32060 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:37.165808 32244 consensus_queue.cc:1046] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260806 20:15:37.288439 30869 raft_consensus.cc:1226] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:37.237439 32101 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.414s user 0.011s sys 0.025s
I20260806 20:15:37.176219 32231 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:37.187438 32229 consensus_meta.cc:287] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.217s user 0.018s sys 0.002s
I20260806 20:15:37.233234 31415 raft_consensus.cc:1226] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:37.283449 32218 consensus_meta.cc:287] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.072s user 0.017s sys 0.006s
W20260806 20:15:37.290756 32224 consensus_meta.cc:287] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.156s user 0.013s sys 0.009s
I20260806 20:15:36.941421 32271 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.324s
I20260806 20:15:37.006388 32103 consensus_queue.cc:237] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.238686 32270 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:37.282824 32061 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.191612 32046 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:37.297520 32254 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.298312 30708 raft_consensus.cc:595] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Received request to transfer leadership to TS 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:37.256172 32236 consensus_queue.cc:260] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.304627 32127 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.306250 32274 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 20:15:37.309135 32279 log.cc:924] Time spent T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260806 20:15:37.310292 32060 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:37.311833 32284 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:37.255394 31416 rpcz_store.cc:276] 0806 20:15:35.831502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:35.831674 (+ 172us) service_pool.cc:227] Handling call
0806 20:15:35.831992 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:35.881013 (+ 49021us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:37.027281 (+1146268us) spinlock_profiling.cc:243] Waited 9.89 ms on lock 0x7b5400570b54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:37.027773 (+ 492us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22736128}
I20260806 20:15:37.214002 30863 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:37.435166 32286 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:37.437902 31417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 371) took 1357 ms. Trace:
W20260806 20:15:37.240430 32217 consensus_meta.cc:287] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.126s user 0.021s sys 0.000s
I20260806 20:15:37.240125 32277 raft_consensus.cc:729] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:15:37.331012 32101 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.521s user 0.012s sys 0.025s
I20260806 20:15:37.353650 32218 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:37.406530 32046 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.431067 30707 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:37.440183 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1371) took 1272 ms. Trace:
I20260806 20:15:37.430464 32236 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:37.442273 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 280807us
I20260806 20:15:37.347790 32229 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.463549 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:37.430308 30784 raft_consensus.cc:2379] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 2: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:37.466470 30868 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:37.472990 30788 rpcz_store.cc:276] 0806 20:15:36.154093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.154250 (+ 157us) service_pool.cc:227] Handling call
0806 20:15:36.168534 (+ 14284us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:36.610537 (+442003us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:37.425730 (+815193us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54001d1ad4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:37.426102 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":933587840}
I20260806 20:15:37.455454 32286 raft_consensus.cc:686] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:37.487550 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:37.488306 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 991) took 1186 ms. Trace:
I20260806 20:15:37.438836 31417 rpcz_store.cc:276] 0806 20:15:36.080700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.089619 (+ 8919us) service_pool.cc:227] Handling call
0806 20:15:36.146857 (+ 57238us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:36.820664 (+673807us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:37.437277 (+616613us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b540058f2d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:37.437777 (+ 500us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1377057536}
W20260806 20:15:37.484094 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 305114us
I20260806 20:15:37.258610 32119 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.624241 30865 raft_consensus.cc:1226] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:37.610250 30867 rpcz_store.cc:276] 0806 20:15:36.287427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.291552 (+ 4125us) service_pool.cc:227] Handling call
0806 20:15:36.292240 (+ 688us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:36.292765 (+ 525us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:36.602384 (+309619us) log.cc:842] Serialized 14 byte log entry
0806 20:15:37.473236 (+870852us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:37.473312 (+ 76us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:37.473388 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:37.473524 (+ 136us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:37.473743 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56735,"prepare.queue_time_us":404958,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":71,"raft.queue_time_us":289006,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":26054,"thread_start_us":944666,"threads_started":3,"wal-append.queue_time_us":605355}
I20260806 20:15:37.443734 32060 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.649235 32135 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:37.622522 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1010) took 1538 ms. Trace:
I20260806 20:15:37.712361 30707 rpcz_store.cc:276] 0806 20:15:36.083923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.088394 (+ 4471us) service_pool.cc:227] Handling call
0806 20:15:37.430172 (+1341778us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:37.430680 (+ 508us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.492985 (+ 62305us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:37.622402 (+129417us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48337152}
I20260806 20:15:37.714066 32286 consensus_queue.cc:237] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.692374 32257 consensus_queue.cc:1035] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.390s
I20260806 20:15:37.719013 31416 raft_consensus.cc:1226] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:37.609347 32285 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:37.669543 32220 consensus_meta.cc:287] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.508s user 0.021s sys 0.002s
W20260806 20:15:37.723310 30754 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.013s sys 0.028s
I20260806 20:15:37.679548 30757 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:37.741037 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 61675.3us
I20260806 20:15:37.692077 30863 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:37.660307 32127 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:37.744148 32127 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.746207 30863 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 20:15:37.722854 30676 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.011s sys 0.030s
W20260806 20:15:37.749254 30676 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.012s sys 0.030s
I20260806 20:15:37.652426 32252 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:37.751065 32220 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:37.754880 32245 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 1.330s user 0.009s sys 0.000s
W20260806 20:15:37.739934 30754 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.546s user 0.014s sys 0.028s
W20260806 20:15:37.756851 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 568195us
I20260806 20:15:37.698208 31414 raft_consensus.cc:1226] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:37.759936 30863 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:37.761085 32246 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 20:15:37.706900 32130 consensus_meta.cc:287] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.592s user 0.013s sys 0.012s
W20260806 20:15:37.737560 32221 consensus_meta.cc:287] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.128s user 0.020s sys 0.000s
I20260806 20:15:37.771180 32220 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:37.699410 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:37.786684 30834 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.007s sys 0.009s
W20260806 20:15:37.804235 30834 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.007s sys 0.009s
I20260806 20:15:37.804536 30676 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:37.805176 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 623658us
W20260806 20:15:37.807175 31419 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.008s sys 0.016s
W20260806 20:15:37.807889 31419 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.600s user 0.008s sys 0.016s
I20260806 20:15:37.770771 32252 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:37.818686 30756 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.011s sys 0.015s
W20260806 20:15:37.819283 30756 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.585s user 0.011s sys 0.015s
W20260806 20:15:37.812590 30833 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.009s sys 0.017s
I20260806 20:15:37.761329 31414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 368) took 2076 ms. Trace:
W20260806 20:15:37.824590 30833 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.586s user 0.009s sys 0.017s
I20260806 20:15:37.807423 30707 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:37.803181 32296 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:37.820453 32078 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.034s sys 0.031s
I20260806 20:15:37.810034 32257 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:37.822947 32203 consensus_meta.cc:287] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.041s user 0.014s sys 0.006s
I20260806 20:15:37.870314 30783 raft_consensus.cc:1226] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:37.870937 32287 consensus_queue.cc:1046] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.107s
I20260806 20:15:37.739848 32135 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.680020 32284 consensus_queue.cc:1046] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20260806 20:15:37.833253 32026 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:37.871316 32127 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.829798 32101 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:37.836670 31829 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:37.795699 32130 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:37.868593 32078 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.557s user 0.034s sys 0.032s
I20260806 20:15:37.771821 32195 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:37.882407 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 76465us
I20260806 20:15:37.795519 32221 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.665107 32277 consensus_queue.cc:260] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:37.894734 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 90372.8us
I20260806 20:15:37.765534 32285 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.834414 32252 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.896719 31421 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:37.898633 30867 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:37.900111 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:37.900046 30676 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:37.901031 30788 raft_consensus.cc:2379] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:37.883311 32217 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:37.884262 32061 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:37.904739 31416 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:37.889531 32254 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:37.897073 32148 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:37.892799 32229 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.832031 31414 rpcz_store.cc:276] 0806 20:15:35.684791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:35.703726 (+ 18935us) service_pool.cc:227] Handling call
0806 20:15:35.711666 (+ 7940us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.759791 (+2048125us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:37.760843 (+ 1052us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54005d2854. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:37.761233 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4490072448}
I20260806 20:15:37.875293 32203 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.899308 32224 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:37.901412 32195 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:37.915112 30863 raft_consensus.cc:1261] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 20:15:37.900519 32060 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.868131 32296 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.881525 32026 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.919826 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:37.925700 32293 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:37.902980 32046 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:37.886945 32101 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.908535 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:37.887374 31829 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.901109 32119 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:37.875077 32257 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.976433 31423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 247) took 1957 ms. Trace:
I20260806 20:15:37.976853 31423 rpcz_store.cc:276] 0806 20:15:36.108251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.117300 (+ 9049us) service_pool.cc:227] Handling call
0806 20:15:37.975703 (+1858403us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:37.975938 (+ 235us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.976117 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:37.976141 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:37.976178 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:37.976202 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:37.976267 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:37.976376 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3159424}
I20260806 20:15:37.977576 31829 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:37.978257 32293 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:37.980144 30804 tablet_copy_source_session.cc:205] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:37.978587 32257 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.980747 32293 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.912581 32277 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:37.952873 32306 consensus_queue.cc:1046] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 20:15:37.953687 30871 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 994) took 1417 ms (client timeout 1651 ms). Trace:
I20260806 20:15:37.962738 31422 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:37.931694 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:37.905207 32220 consensus_queue.cc:1046] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260806 20:15:37.912379 32217 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:37.966208 32135 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.968015 32130 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.917311 32061 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:37.997417 30871 rpcz_store.cc:269] 0806 20:15:36.536537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.536899 (+ 362us) service_pool.cc:227] Handling call
0806 20:15:37.953431 (+1416532us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b54000b1154. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:37.953615 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49035136}
I20260806 20:15:37.927245 32148 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.930452 32254 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.001819 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:38.033931 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 995) took 1480 ms. Trace:
I20260806 20:15:38.040148 31423 raft_consensus.cc:2379] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:37.970435 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1205) took 1074 ms. Trace:
I20260806 20:15:38.043257 30865 rpcz_store.cc:276] 0806 20:15:36.553662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.553918 (+ 256us) service_pool.cc:227] Handling call
0806 20:15:36.595569 (+ 41651us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.625100 (+1029531us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.033689 (+408589us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540014bc54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:38.033870 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2353867520}
I20260806 20:15:37.951232 32224 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:37.983495 32101 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:37.981675 32292 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:37.984611 32290 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:38.023895 31422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 376) took 1908 ms. Trace:
I20260806 20:15:38.050855 30834 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.030833 32047 raft_consensus.cc:3032] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:37.998909 30708 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:38.032457 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1301) took 1181 ms. Trace:
I20260806 20:15:38.034993 32227 raft_consensus.cc:3032] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:37.981220 32282 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:38.008539 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:37.981313 30804 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:53732 (request call id 46) took 2434 ms. Trace:
I20260806 20:15:37.981518 32301 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.057416 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:38.059904 31422 rpcz_store.cc:276] 0806 20:15:36.115151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.115338 (+ 187us) service_pool.cc:227] Handling call
0806 20:15:37.961627 (+1846289us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:37.961903 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:38.023545 (+ 61642us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.023817 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4863360}
W20260806 20:15:38.060948 32289 log.cc:924] Time spent T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
I20260806 20:15:38.134028 32290 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.059667 32047 raft_consensus.cc:729] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:38.056635 32221 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:38.063246 32225 consensus_queue.cc:260] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.064404 32252 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:38.064726 32217 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:37.981602 32295 consensus_queue.cc:1035] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.227s
I20260806 20:15:38.070366 30709 rpcz_store.cc:276] 0806 20:15:36.851448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.856785 (+ 5337us) service_pool.cc:227] Handling call
0806 20:15:38.031629 (+1174844us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:38.031894 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:38.032095 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:38.032123 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:38.032169 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:38.032209 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.032269 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:38.032395 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57279872}
W20260806 20:15:38.061800 32138 consensus_meta.cc:287] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.292s user 0.014s sys 0.008s
W20260806 20:15:38.055691 30757 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.164s (SENT) [suppressed 22 similar messages]
I20260806 20:15:38.136884 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:38.065371 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 997) took 1443 ms. Trace:
I20260806 20:15:38.066213 32292 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.135787 32047 consensus_queue.cc:260] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:38.140637 32047 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:38.141625 32292 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:38.087124 32227 raft_consensus.cc:729] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:38.086844 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:38.142992 30871 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:38.089874 30804 rpcz_store.cc:276] 0806 20:15:35.553130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:35.562279 (+ 9149us) service_pool.cc:227] Handling call
0806 20:15:37.979633 (+2417354us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54002ef020. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71515ff730 00007f7151679305 00007f715167924e 00007f715658c5e6 00007f7156595e5e 00007f714e07c5db 00007f7156595dda 00007f7156590eec 00007f715658c24b 00007f7156580e45
0806 20:15:37.981226 (+ 1593us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":929694976}
I20260806 20:15:38.091109 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:38.091818 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:38.059103 32270 consensus_meta.cc:287] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.652s user 0.018s sys 0.000s
I20260806 20:15:38.154958 32270 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.157778 30713 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:38.158581 32302 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:37.981771 32283 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:38.133309 32149 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:38.045234 30863 rpcz_store.cc:276] 0806 20:15:36.896481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:37.862997 (+966516us) service_pool.cc:227] Handling call
0806 20:15:37.890090 (+ 27093us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.950799 (+ 60709us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:37.964129 (+ 13330us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:38.056116 32291 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:38.056612 32010 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.136694 32290 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:38.165097 32010 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.134130 32307 consensus_queue.cc:1046] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260806 20:15:38.167778 32148 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:38.168588 32203 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:38.165691 31416 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:38.169847 32285 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:38.137897 30869 rpcz_store.cc:276] 0806 20:15:36.622292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.686792 (+ 64500us) service_pool.cc:227] Handling call
0806 20:15:36.687648 (+ 856us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.289428 (+601780us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.065128 (+775700us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b540014c8d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:38.065311 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1380874624}
I20260806 20:15:38.063248 32026 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:38.137291 32138 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.139678 32225 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:38.088745 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.092985 32284 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260806 20:15:38.165119 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:38.119266 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:38.176658 30754 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:38.157254 32270 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:38.159150 32302 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.183355 30703 raft_consensus.cc:1226] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:38.184535 32122 raft_consensus.cc:3032] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:38.187213 32224 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:38.136693 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:38.197867 32026 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:38.198582 32302 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:38.198746 30863 raft_consensus.cc:1226] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:38.195079 32122 raft_consensus.cc:729] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:38.209662 32026 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:38.201241 32247 raft_consensus.cc:3032] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:38.201321 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1013) took 1880 ms. Trace:
I20260806 20:15:38.086747 32282 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.210265 32026 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:38.210443 32247 raft_consensus.cc:729] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:38.198530 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:38.203248 30711 raft_consensus.cc:3032] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 20:15:38.203342 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 152565us
I20260806 20:15:38.180352 32283 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.204010 32130 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:38.211191 30703 rpcz_store.cc:276] 0806 20:15:36.321071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.182461 (+1861390us) service_pool.cc:227] Handling call
0806 20:15:38.183076 (+ 615us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:38.199750 (+ 16674us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.201253 (+ 1503us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:38.219100 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:38.204622 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.207895 32295 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.208488 32217 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.211223 30711 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:38.193889 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.213831 30834 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.210681 32122 consensus_queue.cc:260] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.144536 32296 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:38.223631 30865 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:38.210278 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 74) took 1030 ms. Trace:
W20260806 20:15:38.205833 30864 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 986) took 1935 ms (client timeout 1536 ms). Trace:
I20260806 20:15:38.254258 30834 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.256397 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.257174 30711 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.252516 30863 rpcz_store.cc:276] 0806 20:15:37.180340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.198207 (+1017867us) service_pool.cc:227] Handling call
0806 20:15:38.198545 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:38.209974 (+ 11429us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.210216 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:38.250305 32217 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.299388 32061 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:38.299924 30711 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:38.248526 30707 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:38.298187 30834 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.257853 32122 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:38.261603 32220 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:38.259526 32234 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:38.257198 30703 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:38.260524 32283 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:38.261178 32303 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:38.319626 30834 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.236409 32149 consensus_queue.cc:1046] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260806 20:15:38.326507 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.327077 32149 raft_consensus.cc:3032] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:38.327591 32149 raft_consensus.cc:729] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:38.328596 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.304446 32254 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:38.307102 32257 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 20:15:38.255028 30864 rpcz_store.cc:269] 0806 20:15:36.265061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.315441 (+ 50380us) service_pool.cc:227] Handling call
0806 20:15:38.091654 (+1776213us) spinlock_profiling.cc:243] Waited 9.21 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:38.199672 (+108018us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21174656}
I20260806 20:15:38.257076 32291 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.067s
I20260806 20:15:38.306105 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:38.172783 32301 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:38.324596 30870 consensus_meta.cc:287] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.848s user 0.013s sys 0.001s
I20260806 20:15:38.320168 32234 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:38.236774 32295 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.336499 32089 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:38.328785 32149 consensus_queue.cc:260] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.335489 30834 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.332775 32138 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:38.333789 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:38.341250 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:38.350363 32217 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:38.350944 30861 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:38.218498 32247 consensus_queue.cc:260] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.360558 32285 raft_consensus.cc:3032] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:38.352949 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.359826 32089 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:38.360864 32234 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:38.352432 30870 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:38.329267 30783 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
W20260806 20:15:38.371779 32236 consensus_meta.cc:287] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.883s user 0.013s sys 0.001s
I20260806 20:15:38.357033 30871 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:38.365479 32244 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:38.376071 30861 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 70) took 2080 ms (client timeout 2329 ms). Trace:
I20260806 20:15:38.371567 30708 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:38.373488 32285 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:38.378578 32149 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:38.377170 32089 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:38.379969 32219 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.356659 30711 raft_consensus.cc:1261] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260806 20:15:38.379117 32280 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:38.408586 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 508549us
I20260806 20:15:38.407105 30711 pending_rounds.cc:76] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 1
W20260806 20:15:38.401436 31419 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.006s sys 0.009s
W20260806 20:15:38.408035 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 354132us
I20260806 20:15:38.380501 32247 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:38.407239 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:38.409086 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 73) took 1229 ms. Trace:
I20260806 20:15:38.409616 30676 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.410527 30792 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:38.412169 31419 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.007s sys 0.009s
I20260806 20:15:38.380657 32244 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:38.412847 31419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 385) took 1270 ms. Trace:
I20260806 20:15:38.380954 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.418673 30757 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:38.389679 30707 raft_consensus.cc:2379] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:38.413095 31419 rpcz_store.cc:276] 0806 20:15:37.142868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:37.203502 (+ 60634us) service_pool.cc:227] Handling call
0806 20:15:37.204151 (+ 649us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.207392 (+ 3241us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:15:37.811210 (+603818us) log.cc:842] Serialized 14 byte log entry
0806 20:15:38.412520 (+601310us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:38.412583 (+ 63us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:38.412621 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.412692 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:38.412801 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":781573,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":70451,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":6317,"thread_start_us":1201878,"threads_started":2}
I20260806 20:15:38.418675 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 73) took 2206 ms. Trace:
I20260806 20:15:38.414171 30711 pending_rounds.cc:97] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:15:38.447436 30711 raft_consensus.cc:2866] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:38.448143 30712 rpcz_store.cc:276] 0806 20:15:36.211998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.390298 (+2178300us) service_pool.cc:227] Handling call
0806 20:15:38.418064 (+ 27766us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:38.418283 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:38.418308 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:38.418348 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:38.418377 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.418428 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:38.418537 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:38.401022 32287 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.003s sys 0.013s
I20260806 20:15:38.451767 32235 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:38.452570 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1303) took 1592 ms. Trace:
I20260806 20:15:38.419714 32267 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.420229 32244 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:38.451403 32122 consensus_queue.cc:1046] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
W20260806 20:15:38.478281 30789 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.015s sys 0.052s
I20260806 20:15:38.411891 32282 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:38.359872 32303 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.414232 32219 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.406756 31420 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 373) took 2307 ms. Trace:
W20260806 20:15:38.381335 30861 rpcz_store.cc:269] 0806 20:15:36.295880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.316041 (+ 20161us) service_pool.cc:227] Handling call
0806 20:15:38.299743 (+1983702us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:38.373497 (+ 73754us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9873152}
W20260806 20:15:38.502871 30789 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.015s sys 0.052s
I20260806 20:15:38.531741 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:38.480387 30711 rpcz_store.cc:276] 0806 20:15:36.859843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.880993 (+ 21150us) service_pool.cc:227] Handling call
0806 20:15:36.897155 (+ 16162us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:38.203150 (+1305995us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5400109da0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71515ff730 00007f7151671300 00007f7151672371 00007f715166eedf 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc
0806 20:15:38.452304 (+249154us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.452510 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":276037504}
I20260806 20:15:38.763784 30792 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:38.515235 31419 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:38.583385 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:38.818995 30792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1379) took 1907 ms. Trace:
W20260806 20:15:38.684609 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 216060us
W20260806 20:15:38.654166 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 142484us
I20260806 20:15:38.818279 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:38.819371 30792 rpcz_store.cc:276] 0806 20:15:36.911564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.911762 (+ 198us) service_pool.cc:227] Handling call
0806 20:15:38.407055 (+1495293us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:38.818937 (+411882us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18322304}
W20260806 20:15:38.693856 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 284303us
W20260806 20:15:38.652967 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 138564us
I20260806 20:15:38.820703 30835 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:38.821039 30834 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.820760 30676 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:38.577021 32314 log.cc:924] Time spent T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20260806 20:15:38.451879 32286 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:15:38.637457 32313 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
W20260806 20:15:38.536756 31418 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 2.114s user 0.020s sys 0.015s
I20260806 20:15:38.826022 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:38.832818 32286 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.833063 32130 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:38.835927 32130 raft_consensus.cc:686] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:38.836802 32130 consensus_queue.cc:237] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:38.837860 32316 log.cc:924] Time spent T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.303s user 0.000s sys 0.001s
W20260806 20:15:38.835073 31418 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 2.412s user 0.025s sys 0.023s
I20260806 20:15:38.828754 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:38.842995 30788 raft_consensus.cc:3032] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:38.843283 30788 raft_consensus.cc:729] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:38.824229 32242 consensus_queue.cc:1035] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260806 20:15:38.844023 30788 consensus_queue.cc:260] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.827131 32257 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.844913 30788 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:38.844959 32257 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.827620 32283 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:38.828163 30676 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:38.831422 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1383) took 1371 ms. Trace:
W20260806 20:15:38.783743 32287 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.912s user 0.003s sys 0.017s
I20260806 20:15:38.450464 32236 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:38.637881 32295 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:38.841917 32240 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:38.573326 32277 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.744019 32122 raft_consensus.cc:3032] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:38.534626 32285 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.863668 32122 raft_consensus.cc:729] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:15:38.865056 32220 consensus_meta.cc:287] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.543s user 0.007s sys 0.002s
I20260806 20:15:38.795344 31420 rpcz_store.cc:276] 0806 20:15:36.081274 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.088534 (+ 7260us) service_pool.cc:227] Handling call
0806 20:15:38.360122 (+2271588us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:38.367436 (+ 7314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:38.375414 (+ 7978us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:38.378018 (+ 2604us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:38.379168 (+ 1150us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:38.381536 (+ 2368us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.383939 (+ 2403us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:38.388151 (+ 4212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5381504}
I20260806 20:15:38.731335 32254 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.795591 32217 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:38.538765 32235 raft_consensus.cc:686] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:38.630697 32089 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.771018 32233 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:38.858875 30792 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:38.817420 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.849143 30789 rpcz_store.cc:276] 0806 20:15:37.460380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:37.460602 (+ 222us) service_pool.cc:227] Handling call
0806 20:15:37.466305 (+ 5703us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.466717 (+ 412us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:37.885375 (+418658us) log.cc:842] Serialized 14 byte log entry
0806 20:15:38.831100 (+945725us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:38.831151 (+ 51us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:38.831188 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.831261 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:38.831369 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14606,"prepare.queue_time_us":506242,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":16671,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":1323304,"threads_started":2,"wal-append.queue_time_us":948287}
I20260806 20:15:38.539784 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.418221 32280 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.485268 32267 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.864589 32122 consensus_queue.cc:260] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.870245 32257 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:38.871325 32295 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.514894 32301 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:38.857982 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.911782 30711 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:38.865532 32220 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:38.832986 32250 consensus_queue.cc:1046] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260806 20:15:38.847589 32149 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:38.867054 32325 ts_tablet_manager.cc:927] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
I20260806 20:15:38.857283 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1169) took 3250 ms. Trace:
I20260806 20:15:38.882684 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:38.919685 30792 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:38.921304 30835 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:38.870429 32277 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.884577 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:38.869606 32291 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:38.905884 31423 raft_consensus.cc:2396] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:38.907627 32308 raft_consensus.cc:3032] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:38.868960 30676 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.910627 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1389) took 1077 ms. Trace:
I20260806 20:15:38.914192 30834 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:38.891709 32233 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.869988 32240 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.957420 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.884586 32217 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:38.921404 30703 raft_consensus.cc:2379] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:38.923394 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:38.908769 32315 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260806 20:15:38.886457 32103 consensus_queue.cc:1046] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260806 20:15:38.894829 30707 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
W20260806 20:15:38.903101 32234 consensus_meta.cc:287] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.515s user 0.009s sys 0.000s
W20260806 20:15:38.963703 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 49598.8us
I20260806 20:15:38.964823 31415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 372) took 2883 ms. Trace:
I20260806 20:15:38.966161 32297 raft_consensus.cc:3032] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:38.881042 32254 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.960024 32220 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:38.956274 32141 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:38.913380 32295 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.974578 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:38.963353 30790 rpcz_store.cc:276] 0806 20:15:37.834660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:37.837401 (+ 2741us) service_pool.cc:227] Handling call
0806 20:15:37.867192 (+ 29791us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:37.867555 (+ 363us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:37.867582 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:37.877130 (+ 9548us) log.cc:842] Serialized 26 byte log entry
0806 20:15:38.304007 (+426877us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:38.304047 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.304120 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:38.910398 (+606278us) raft_consensus.cc:1671] finished
0806 20:15:38.910418 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:38.910559 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5935,"raft.queue_time_us":54842,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":8830,"thread_start_us":398241,"threads_started":1,"wal-append.queue_time_us":485033}
I20260806 20:15:38.962285 32308 raft_consensus.cc:729] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:38.981871 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:38.959100 32250 consensus_queue.cc:1225] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer 42cabca48db7403b8523a3382a046043 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260806 20:15:38.963514 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 94611.2us
I20260806 20:15:38.990907 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:38.970227 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:38.994462 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:38.882308 32312 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20260806 20:15:38.975608 32287 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:38.975452 32301 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:39.026687 32254 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.005750 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:38.916076 32013 consensus_queue.cc:1046] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
I20260806 20:15:39.003396 32222 raft_consensus.cc:3032] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:38.981117 31415 rpcz_store.cc:276] 0806 20:15:36.080985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.088278 (+ 7293us) service_pool.cc:227] Handling call
0806 20:15:36.138564 (+ 50286us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.346283 (+1207719us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.964533 (+1618250us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540067f2d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:38.964754 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2416823680}
W20260806 20:15:39.025981 32026 consensus_meta.cc:287] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.797s user 0.013s sys 0.000s
I20260806 20:15:38.988780 32103 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:39.031965 30676 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.032128 32103 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.030946 32222 raft_consensus.cc:729] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:39.032759 32312 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:39.033361 32312 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:39.033631 32291 consensus_queue.cc:1046] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:15:39.034436 30676 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:38.982985 32234 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.035636 32222 consensus_queue.cc:260] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.036526 32234 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.036676 32312 consensus_queue.cc:1035] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:38.925393 32047 consensus_queue.cc:1046] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20260806 20:15:39.037958 32283 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:39.034744 32291 raft_consensus.cc:3032] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:39.038412 32283 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:39.036186 32138 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.038816 32283 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.039309 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.040010 30790 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:39.040351 32291 raft_consensus.cc:729] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:39.040880 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.038795 32222 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.031669 32026 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:39.043502 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1008) took 2960 ms. Trace:
I20260806 20:15:38.983327 32235 consensus_queue.cc:237] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:38.983271 32297 raft_consensus.cc:729] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:39.045091 32297 consensus_queue.cc:260] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.041229 32291 consensus_queue.cc:260] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:39.045969 32026 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:39.046708 32291 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.046936 32026 raft_consensus.cc:3032] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:39.047484 32026 raft_consensus.cc:729] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:39.030968 32308 consensus_queue.cc:260] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:39.048151 32138 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:39.048036 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.049418 30707 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:39.050439 32308 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.051168 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.051960 30707 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:39.039649 32240 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.030539 32301 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:38.975396 32149 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:38.981875 32122 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.059726 30835 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:39.064746 30834 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:39.065496 30834 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:39.065876 30788 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:39.062657 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.037993 30676 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.036453 32103 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:39.067312 30789 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:39.074976 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.069049 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.079074 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.080051 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:38.981119 32257 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:39.082016 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.083016 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:39.084025 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.070981 30835 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.085212 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:38.993001 32220 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:39.088774 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 50856us
I20260806 20:15:39.082788 30711 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:39.068487 31957 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:39.057104 32280 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:39.055383 32217 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:39.056430 32233 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:39.136911 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:39.137619 30711 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.128837 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.053310 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.058076 32301 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:39.129402 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:39.069308 32240 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:39.092077 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 194220us
I20260806 20:15:39.094091 30834 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:39.094174 30789 raft_consensus.cc:2379] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:39.090852 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.058598 32250 raft_consensus.cc:3032] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:39.005059 32141 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:39.149147 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.128279 31957 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:38.922052 30786 rpcz_store.cc:276] 0806 20:15:35.576880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:35.672632 (+ 95752us) service_pool.cc:227] Handling call
0806 20:15:35.672947 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:36.117227 (+444280us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.797406 (+2680179us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b54001702d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:38.827358 (+ 29952us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":688683776}
I20260806 20:15:39.133050 32026 consensus_queue.cc:260] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.132926 32089 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:39.134486 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 253199us
W20260806 20:15:39.091251 30708 consensus_meta.cc:287] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.696s user 0.007s sys 0.003s
I20260806 20:15:39.084532 32234 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:39.106058 32220 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.090467 32242 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:39.067226 32149 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.093297 32236 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:39.149848 30789 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:39.239042 32130 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.158021 30757 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:39.146255 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 52260.5us
I20260806 20:15:39.214816 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1396) took 1009 ms. Trace:
I20260806 20:15:39.146023 32079 raft_consensus.cc:3032] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:39.276206 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:39.215547 32217 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:39.043892 30705 rpcz_store.cc:276] 0806 20:15:36.082693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.089492 (+ 6799us) service_pool.cc:227] Handling call
0806 20:15:37.903205 (+1813713us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71bbce 000000000034aea9 00007f715653ba30 00007f71566017b4 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:39.042727 (+1139522us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71bc6c 00007f714e71bbd6 000000000034aea9 00007f715653ba30 00007f71566017b4 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:39.042866 (+ 139us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.043044 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:39.043072 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:39.043212 (+ 140us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:39.043247 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.043304 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:39.043440 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11460096}
I20260806 20:15:39.319413 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.146183 32240 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:39.319948 30709 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:39.320276 32240 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.319823 32297 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.241994 31957 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:39.132185 32257 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.330881 32238 raft_consensus.cc:3032] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:39.331333 32238 raft_consensus.cc:729] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:39.326128 32130 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.332221 32238 consensus_queue.cc:260] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:39.333873 32130 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.334681 30705 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:39.336587 32238 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:38.410511 30869 rpcz_store.cc:276] 0806 20:15:37.179787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.170700 (+990913us) service_pool.cc:227] Handling call
0806 20:15:38.171028 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:38.171258 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:15:38.171564 (+ 306us) log.cc:842] Serialized 14 byte log entry
0806 20:15:38.408781 (+237217us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:38.408833 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:38.408873 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:38.408935 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:38.409036 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":594,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":3824,"raft.queue_time_us":7860,"thread_start_us":229124,"threads_started":1}
W20260806 20:15:39.345542 31917 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260806 20:15:39.314745 32330 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:39.350091 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1387) took 1884 ms. Trace:
I20260806 20:15:39.350464 30787 rpcz_store.cc:276] 0806 20:15:37.470087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:37.482707 (+ 12620us) service_pool.cc:227] Handling call
0806 20:15:39.349375 (+1866668us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:39.349587 (+ 212us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.349768 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:39.349795 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:39.349831 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:39.349861 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.349933 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:39.350037 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":257395968}
I20260806 20:15:39.352315 30787 raft_consensus.cc:1226] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:39.352890 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1405) took 1028 ms. Trace:
I20260806 20:15:39.353783 30787 rpcz_store.cc:276] 0806 20:15:38.324337 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.351494 (+1027157us) service_pool.cc:227] Handling call
0806 20:15:39.351809 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.352647 (+ 838us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.352814 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:39.360065 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1010) took 2529 ms. Trace:
I20260806 20:15:39.360873 30787 raft_consensus.cc:1226] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:39.360531 30867 rpcz_store.cc:276] 0806 20:15:36.830847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:37.741038 (+910191us) service_pool.cc:227] Handling call
0806 20:15:37.741738 (+ 700us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.905500 (+163762us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.359786 (+1454286us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54000b0254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:39.359995 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":353575680}
W20260806 20:15:39.304859 30756 connection.cc:232] RPC call timeout handler was delayed by -1.8386010190006346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
I20260806 20:15:39.146482 30707 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:39.240381 32233 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:39.308269 30835 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:39.275599 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 127271us
I20260806 20:15:39.378180 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.216310 32138 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:39.308915 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 162811us
I20260806 20:15:39.308576 30788 rpcz_store.cc:276] 0806 20:15:38.205666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.090584 (+884918us) service_pool.cc:227] Handling call
0806 20:15:39.214735 (+124151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:39.302065 32234 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:39.325762 32329 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.355515 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:39.282516 30708 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:39.146234 32250 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:39.306949 32026 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.304124 32325 tablet_copy_client.cc:316] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Beginning tablet copy session from remote peer at address 127.28.186.193:42937
I20260806 20:15:39.380457 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1404) took 1063 ms. Trace:
W20260806 20:15:39.382017 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 73839.8us
I20260806 20:15:39.313959 32079 raft_consensus.cc:729] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:15:39.417546 30706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1302) took 2561 ms (client timeout 1740 ms). Trace:
I20260806 20:15:39.363377 32236 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.423031 30790 rpcz_store.cc:276] 0806 20:15:38.317077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.327474 (+1010397us) service_pool.cc:227] Handling call
0806 20:15:39.379946 (+ 52472us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.380132 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:39.380157 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:39.380210 (+ 53us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:39.380239 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.380290 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:39.380396 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:39.416772 32247 consensus_meta.cc:287] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.999s user 0.005s sys 0.008s
W20260806 20:15:39.421622 30706 rpcz_store.cc:269] 0806 20:15:36.855791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.856041 (+ 250us) service_pool.cc:227] Handling call
0806 20:15:39.355359 (+2499318us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:39.417468 (+ 62109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31036160}
I20260806 20:15:39.428944 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:39.391319 30711 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:39.414017 31917 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:39.414063 32234 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:39.414535 32251 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:39.375960 32233 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:39.410662 32293 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.327611 32277 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:39.331661 32257 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.432192 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:39.432416 30834 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.460108 30711 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.463188 30871 raft_consensus.cc:1226] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:39.435598 30676 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.433745 32295 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.436266 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.328132 32217 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.466739 32217 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.445508 32233 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.425413 30786 raft_consensus.cc:1261] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:39.450750 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:39.451601 32324 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.846s user 0.011s sys 0.008s
I20260806 20:15:39.451483 32079 consensus_queue.cc:260] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:39.452117 32330 consensus_queue.cc:1046] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:39.424993 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1312) took 1525 ms. Trace:
I20260806 20:15:39.442304 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.417217 32329 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.442863 32251 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:39.468199 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:39.447024 32247 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:39.453606 32293 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.472066 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.478418 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1304) took 2617 ms. Trace:
I20260806 20:15:39.589085 30713 rpcz_store.cc:276] 0806 20:15:36.861092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:37.135547 (+274455us) service_pool.cc:227] Handling call
0806 20:15:37.136358 (+ 811us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:38.158215 (+1021857us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.478139 (+1319924us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400090bd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:39.478354 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2348372864}
I20260806 20:15:39.573585 30787 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:39.467458 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:39.466610 31414 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:39.577493 30867 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:39.471547 32274 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.428267 32236 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.443188 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.470300 30722 tablet_copy_service.cc:140] P 42cabca48db7403b8523a3382a046043: Received BeginTabletCopySession request for tablet aa7cfe759ca64ca89a0529c2d3da5bf7 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 ({username='slave'} at 127.0.0.1:41700)
I20260806 20:15:39.471069 32240 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:39.590803 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 149771us
I20260806 20:15:39.585795 32013 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.468824 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:39.595886 30706 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:39.530256 32257 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election won for term 2
W20260806 20:15:39.605055 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 150537us
I20260806 20:15:39.586297 32251 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.606654 30711 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
W20260806 20:15:39.588398 32323 log.cc:924] Time spent T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20260806 20:15:39.607971 30835 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:39.605830 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 170304us
I20260806 20:15:39.612244 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.612798 30711 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:39.603964 32138 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:39.613719 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.614513 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:39.597738 32103 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.631492 30706 raft_consensus.cc:2379] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:39.612003 30676 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.632912 30834 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.595650 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1397) took 1380 ms. Trace:
I20260806 20:15:39.630330 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.630326 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:39.616995 32257 raft_consensus.cc:686] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:15:39.664016 32222 consensus_meta.cc:287] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.622s user 0.009s sys 0.000s
I20260806 20:15:39.576809 30708 rpcz_store.cc:276] 0806 20:15:37.896888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.318145 (+421257us) service_pool.cc:227] Handling call
0806 20:15:39.422373 (+1104228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:39.609331 32079 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:39.668347 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 46628.9us
I20260806 20:15:39.671823 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.673282 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.630661 30863 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:39.677356 31957 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.678082 32010 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:39.678741 32293 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.666989 30864 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
W20260806 20:15:39.682565 32291 consensus_meta.cc:287] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.635s user 0.010s sys 0.000s
I20260806 20:15:39.683254 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.682440 32240 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.461710 32250 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.604367 32274 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.668040 32242 consensus_queue.cc:1046] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260806 20:15:39.608253 32247 raft_consensus.cc:686] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:39.616437 32089 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:39.674800 30711 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:39.609392 30722 tablet_copy_service.cc:161] P 42cabca48db7403b8523a3382a046043: Beginning new tablet copy session on tablet aa7cfe759ca64ca89a0529c2d3da5bf7 from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 at {username='slave'} at 127.0.0.1:41700: session id = 58f2f6256ee14cb38a1dbcbf5bb590a0-aa7cfe759ca64ca89a0529c2d3da5bf7
W20260806 20:15:39.673697 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 61787us
I20260806 20:15:39.666438 32149 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:39.629897 32341 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:39.607721 30707 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:39.708477 30676 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.630614 31414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 257) took 1706 ms. Trace:
I20260806 20:15:39.616700 32141 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:39.706837 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:39.686110 32301 consensus_meta.cc:287] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.543s user 0.000s sys 0.008s
I20260806 20:15:39.664656 30786 rpcz_store.cc:276] 0806 20:15:38.215034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.283105 (+1068071us) service_pool.cc:227] Handling call
0806 20:15:39.362802 (+ 79697us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.476978 (+114176us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.591688 (+114710us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:39.686076 32138 raft_consensus.cc:2726] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:39.609694 32013 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.703883 31957 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.719415 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.717378 30706 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:39.720702 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:39.715929 32296 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:39.725696 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:39.721673 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.729504 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:39.707350 32089 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:39.722430 32301 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.707275 32217 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:39.753940 30711 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:39.672662 32103 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:39.737902 32283 consensus_meta.cc:287] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.699s user 0.003s sys 0.006s
I20260806 20:15:39.778045 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:39.780905 30870 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 987) took 3515 ms (client timeout 2164 ms). Trace:
W20260806 20:15:39.781339 30870 rpcz_store.cc:269] 0806 20:15:36.265984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.269498 (+ 3514us) service_pool.cc:227] Handling call
0806 20:15:39.780694 (+3511196us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540027fbd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:39.780827 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2409086336}
I20260806 20:15:39.750258 30708 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
W20260806 20:15:39.771754 32244 consensus_meta.cc:287] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.286s user 0.000s sys 0.012s
I20260806 20:15:39.783643 32329 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:39.784283 32122 consensus_meta.cc:287] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.714s user 0.009s sys 0.000s
I20260806 20:15:39.755611 31957 consensus_queue.cc:1035] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260806 20:15:39.764036 32343 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.784809 30787 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:39.769908 32236 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:39.776898 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.786406 30707 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:39.787375 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:39.787600 32267 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.787972 30707 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.753557 32089 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.786159 32286 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:39.783550 32244 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.743026 32138 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:15:39.768975 32308 consensus_meta.cc:287] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.718s user 0.008s sys 0.000s
I20260806 20:15:39.754304 32217 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:39.794945 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:39.772667 32220 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.795603 32283 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:39.797122 30709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41700 (request call id 82) took 1922 ms (client timeout 2087 ms). Trace:
I20260806 20:15:39.795013 32217 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.726020 32257 consensus_queue.cc:237] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.793951 31423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 260) took 1728 ms. Trace:
I20260806 20:15:39.800752 30869 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:39.803145 31423 rpcz_store.cc:276] 0806 20:15:38.090012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.118918 (+ 28906us) service_pool.cc:227] Handling call
0806 20:15:39.793742 (+1674824us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b5400501754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:39.793890 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1645967232}
W20260806 20:15:39.778831 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 59499.1us
I20260806 20:15:39.804986 30676 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.789398 30788 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:39.778275 32141 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:39.795852 30710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1009) took 3712 ms (client timeout 1638 ms). Trace:
I20260806 20:15:39.802575 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:39.805989 30710 rpcz_store.cc:269] 0806 20:15:36.083609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.096421 (+ 12812us) service_pool.cc:227] Handling call
0806 20:15:37.721434 (+1625013us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71bbce 000000000034aea9 00007f715653ba30 00007f71566017b4 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:39.794852 (+2073418us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71bc6c 00007f714e71bbd6 000000000034aea9 00007f715653ba30 00007f71566017b4 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:39.795788 (+ 936us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42414208}
I20260806 20:15:39.807858 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:39.809242 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:39.810623 30835 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:39.811597 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.812372 30711 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:39.799654 32283 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.812649 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.815616 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:39.788930 32234 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.821372 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:39.821682 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:39.821364 30676 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:39.824508 31423 raft_consensus.cc:2379] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 2: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:39.797513 32220 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.817780 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.774078 32274 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:39.813223 32141 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.765543 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:39.801967 32244 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:39.816636 30756 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:39.831236 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:39.831359 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:39.833693 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:39.834968 32236 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:39.825438 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:39.716905 31414 rpcz_store.cc:276] 0806 20:15:37.951422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.453372 (+1501950us) service_pool.cc:227] Handling call
0806 20:15:39.466412 (+ 13040us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.596647 (+130235us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.616922 (+ 20275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:39.836472 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:39.836604 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:39.836694 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:39.829133 31421 raft_consensus.cc:2379] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:39.816346 31957 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:39.837059 30790 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.838670 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:15:39.841216 31389 connection.cc:232] RPC call timeout handler was delayed by -2.4842529880115762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
I20260806 20:15:39.838665 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:39.842547 31389 connection.cc:232] RPC call timeout handler was delayed by -2.4728699970000889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
W20260806 20:15:39.842927 31389 connection.cc:232] RPC call timeout handler was delayed by -2.3162699970125686s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21880000000000002s for a response
W20260806 20:15:39.839152 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 92903.1us
I20260806 20:15:39.828248 30710 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:39.822423 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:39.829074 32138 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:39.843371 31389 connection.cc:232] RPC call timeout handler was delayed by -2.189369997009635s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23290000000000002s for a response
I20260806 20:15:39.844360 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:39.853256 30867 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:39.846846 31388 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:39.821578 32277 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:39.847134 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 78) took 2557 ms. Trace:
I20260806 20:15:39.835404 30756 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:39.855185 30756 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:39.849339 30706 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:39.822461 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:39.822121 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:39.861923 31389 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:39.862351 30832 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:39.856086 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1021) took 2702 ms. Trace:
I20260806 20:15:39.828433 30788 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:39.858662 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:39.785494 32343 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:39.842115 32331 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.860499 30756 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:39.861007 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1022) took 2705 ms. Trace:
I20260806 20:15:39.852524 32138 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:39.854724 30710 rpcz_store.cc:276] 0806 20:15:37.290331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.846360 (+2556029us) service_pool.cc:227] Handling call
0806 20:15:39.846696 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.846846 (+ 150us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:39.846870 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:39.846909 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:39.846934 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.846984 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:39.847086 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:39.865212 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:39.797570 30709 rpcz_store.cc:269] 0806 20:15:37.875113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.300103 (+424990us) service_pool.cc:227] Handling call
0806 20:15:39.796936 (+1496833us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54001b3d54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:39.797045 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":986368768}
I20260806 20:15:39.835451 32236 raft_consensus.cc:686] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:39.864274 30835 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:39.721704 32341 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:39.817140 32247 consensus_queue.cc:237] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.839946 31957 raft_consensus.cc:686] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:39.863318 30867 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:39.869295 30756 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:39.900978 31388 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:39.859875 31414 raft_consensus.cc:2379] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 2: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
W20260806 20:15:39.871071 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 238224us
I20260806 20:15:39.868997 30711 rpcz_store.cc:276] 0806 20:15:37.155232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.834390 (+2679158us) service_pool.cc:227] Handling call
0806 20:15:39.860515 (+ 26125us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.860697 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:39.860721 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:39.860758 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:39.860789 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.860856 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:39.860952 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:39.850131 32227 consensus_queue.cc:260] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:39.862919 30706 rpcz_store.cc:276] 0806 20:15:37.154130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.830331 (+2676201us) service_pool.cc:227] Handling call
0806 20:15:39.848969 (+ 18638us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.855852 (+ 6883us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:39.856041 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:39.864498 32274 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:39.841100 32308 consensus_queue.cc:1035] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260806 20:15:39.901870 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:39.861613 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:39.867386 32231 consensus_queue.cc:1046] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260806 20:15:39.903934 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:39.936825 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 83708us
I20260806 20:15:39.894287 30788 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:39.852610 32027 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260806 20:15:39.941766 30834 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.872097 31418 raft_consensus.cc:2379] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
W20260806 20:15:39.946017 30676 connection.cc:232] RPC call timeout handler was delayed by -1.5914794119889848s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15600000000000003s for a response
I20260806 20:15:39.934499 31417 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:39.902771 30870 raft_consensus.cc:1226] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:39.870855 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:39.867964 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:39.872309 32277 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.991297 30706 raft_consensus.cc:1226] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:39.871917 32138 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:39.863170 32267 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:39.993296 32141 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:39.894732 32344 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:39.891876 32026 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:39.862709 32352 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:39.871941 32220 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:39.933840 32079 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:39.903023 32074 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:39.904627 32283 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 20:15:39.996378 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 92519.6us
I20260806 20:15:39.998937 31389 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.030903 31389 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.032300 31389 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:39.943404 32274 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:40.035379 32274 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:40.036470 32266 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:15:40.000077 30676 connection.cc:232] RPC call timeout handler was delayed by -1.5527794119843747s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20260806 20:15:40.037482 30676 connection.cc:232] RPC call timeout handler was delayed by -1.5194794120034203s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.164s for a response
W20260806 20:15:40.037833 30676 connection.cc:232] RPC call timeout handler was delayed by -1.4582794120069593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1708s for a response
W20260806 20:15:40.004642 32353 log.cc:924] Time spent T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260806 20:15:40.006721 30870 raft_consensus.cc:1226] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:40.015192 31415 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:39.937276 32346 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election won for term 2
W20260806 20:15:40.008679 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 67033us
W20260806 20:15:39.999614 32233 consensus_meta.cc:287] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.527s user 0.006s sys 0.005s
I20260806 20:15:39.946972 32350 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:40.014413 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1025) took 2534 ms. Trace:
I20260806 20:15:40.028604 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1024) took 2548 ms. Trace:
I20260806 20:15:39.999374 32301 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.001752 32345 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:15:40.018741 30784 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1375) took 3166 ms (client timeout 1682 ms). Trace:
I20260806 20:15:40.018726 32236 consensus_queue.cc:237] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.019275 32234 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.015813 32267 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.020357 32303 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:40.019946 31957 consensus_queue.cc:237] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:39.990463 32269 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:40.025820 32352 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.034814 32354 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:40.047442 30676 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:40.016093 32103 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.007323 32343 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:40.021361 32026 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:40.054502 30834 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:39.960799 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.016412 32277 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.037955 32079 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.111953 30722 tablet_copy_source_session.cc:205] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:15:40.113672 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.116463 32217 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:40.112499 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 65146.6us
I20260806 20:15:40.111547 32352 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.132055 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.061868 30706 rpcz_store.cc:276] 0806 20:15:37.480438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.935482 (+2455044us) service_pool.cc:227] Handling call
0806 20:15:39.991117 (+ 55635us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:40.014145 (+ 23028us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:40.014355 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:40.040189 32344 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 20:15:40.120761 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 66367.2us
I20260806 20:15:40.136752 30834 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.050640 32283 raft_consensus.cc:686] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:40.046463 32074 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.120803 30867 raft_consensus.cc:2396] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:40.117905 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1386) took 2653 ms. Trace:
I20260806 20:15:40.114313 32308 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:40.133440 32149 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:40.152580 32337 consensus_queue.cc:1046] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 20:15:40.155534 31420 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:40.155901 32337 raft_consensus.cc:3032] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:40.156255 32337 raft_consensus.cc:729] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:40.156641 32079 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.157006 32149 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:40.157521 32149 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.157928 32283 consensus_queue.cc:237] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.158927 32302 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:15:40.159586 32149 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:40.162544 32277 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.134793 32217 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.048520 32010 consensus_queue.cc:1046] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260806 20:15:40.119171 32267 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:40.164093 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.090085 32270 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:40.122141 32026 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:40.026894 32251 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:40.176119 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 143890us
I20260806 20:15:40.176802 30757 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:40.091336 32346 raft_consensus.cc:686] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:40.063267 32266 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.149027 30867 raft_consensus.cc:1226] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:40.055516 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:40.151079 30783 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1395) took 2169 ms (client timeout 2218 ms). Trace:
I20260806 20:15:40.090091 32007 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260806 20:15:40.127871 32234 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:40.057718 32295 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:40.118227 30784 rpcz_store.cc:269] 0806 20:15:36.852401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.857578 (+ 5177us) service_pool.cc:227] Handling call
0806 20:15:40.017116 (+3159538us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54003d59d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:40.018651 (+ 1535us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":436858112}
I20260806 20:15:40.130038 32341 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.061147 32233 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.175288 32308 raft_consensus.cc:686] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:40.174041 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 992) took 3880 ms. Trace:
I20260806 20:15:40.180785 30676 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:15:40.206213 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 51877.6us
I20260806 20:15:40.143644 32354 raft_consensus.cc:686] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:40.184330 30757 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:40.191054 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.211799 30867 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:40.144220 30786 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:40.192780 30832 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.097357 32350 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.199127 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:40.173709 32303 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:40.173897 32217 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:40.251713 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:40.222916 30832 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:40.205910 30868 rpcz_store.cc:276] 0806 20:15:36.293820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.294137 (+ 317us) service_pool.cc:227] Handling call
0806 20:15:36.338443 (+ 44306us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.489994 (+1151551us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:40.173791 (+2683797us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400376c54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:40.173977 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2573263232}
I20260806 20:15:40.152925 32238 consensus_queue.cc:1046] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260806 20:15:40.189257 32223 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:40.132359 32269 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.250778 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:40.176903 32270 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.263818 30790 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:40.119515 30709 rpcz_store.cc:276] 0806 20:15:37.479900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.933921 (+2454021us) service_pool.cc:227] Handling call
0806 20:15:39.995896 (+ 61975us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:40.008236 (+ 12340us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:40.008268 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:40.009576 (+ 1308us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:40.011711 (+ 2135us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:40.016461 (+ 4750us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:40.027052 (+ 10591us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:40.263460 30871 raft_consensus.cc:1226] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:40.265519 30832 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:40.274657 31389 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:40.276053 31389 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:40.276080 30832 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.201354 32297 consensus_queue.cc:1046] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260806 20:15:40.253657 32103 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.278645 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.210639 32234 raft_consensus.cc:686] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:40.253827 32254 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:40.252076 32074 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.253545 32220 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:40.251096 32344 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:40.213842 32296 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:40.262478 32303 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:40.224550 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:40.222013 31917 consensus_queue.cc:1046] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260806 20:15:40.266695 32217 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.204207 32089 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.273883 31417 raft_consensus.cc:2396] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:40.288000 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:40.267565 32339 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.266148 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:40.212436 32349 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:40.221951 32149 consensus_queue.cc:1046] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20260806 20:15:40.205024 32295 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.273095 32347 raft_consensus.cc:2741] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 20:15:40.294214 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.298501 32303 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.280351 32283 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 20:15:40.292882 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 80205.1us
I20260806 20:15:40.271304 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:40.157032 32337 consensus_queue.cc:260] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:40.295995 32275 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:40.250905 32354 consensus_queue.cc:237] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.292585 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.294634 32296 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:40.196007 30783 rpcz_store.cc:269] 0806 20:15:37.981936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.188223 (+206287us) service_pool.cc:227] Handling call
0806 20:15:40.150863 (+1962640us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54003b29d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:40.151014 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":253992704}
I20260806 20:15:40.305696 32349 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.145196 30791 rpcz_store.cc:276] 0806 20:15:37.464933 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:37.467264 (+ 2331us) service_pool.cc:227] Handling call
0806 20:15:40.017480 (+2550216us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:40.018726 (+ 1246us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:40.019128 (+ 402us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:40.019159 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:40.019672 (+ 513us) log.cc:842] Serialized 26 byte log entry
0806 20:15:40.061471 (+ 41799us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:40.091867 (+ 30396us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:40.096070 (+ 4203us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:40.097351 (+ 1281us) raft_consensus.cc:1671] finished
0806 20:15:40.098643 (+ 1292us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:40.110237 (+ 11594us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7899,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":8620,"spinlock_wait_cycles":268672384,"thread_start_us":28309,"threads_started":1,"wal-append.queue_time_us":39171}
I20260806 20:15:40.321640 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:40.322446 32251 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.354321 32315 raft_consensus.cc:3032] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:40.357239 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:40.307340 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.296396 32344 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.322824 32331 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:40.377034 31421 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:40.375262 32296 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.305018 32341 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:40.319305 32010 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:40.365231 32370 raft_consensus.cc:3032] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:40.322935 32308 consensus_queue.cc:237] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.324198 32149 consensus_queue.cc:1225] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer 58d59beb597f4ed99d84d102ddb41ff5 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:15:40.308864 30864 raft_consensus.cc:1226] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: [2.3-2.3]
I20260806 20:15:40.322692 32269 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.366536 32275 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:40.376075 30784 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
W20260806 20:15:40.368067 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 92050us
I20260806 20:15:40.378149 32315 raft_consensus.cc:729] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:40.397477 32370 raft_consensus.cc:729] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:40.383951 32233 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.384197 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:40.397696 32361 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:40.403965 30832 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.389438 32341 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.379050 32238 consensus_queue.cc:1046] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260806 20:15:40.308967 32346 consensus_queue.cc:237] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.398561 32325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7cfe759ca64ca89a0529c2d3da5bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:15:40.406934 32297 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:40.403079 32275 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.400343 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.418973 30832 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:40.422487 30832 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.423625 30832 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.408180 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.426332 30832 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.414355 32341 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.431001 32370 consensus_queue.cc:260] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.427325 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:40.439718 30832 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:40.385739 32234 consensus_queue.cc:237] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.442026 32142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1969a3911423e8e51fc4f2962515d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 20:15:40.442157 30707 consensus_meta.cc:287] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.654s user 0.013s sys 0.000s
I20260806 20:15:40.427397 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.411864 32270 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:40.444562 30832 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:40.424392 32233 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:40.430033 32257 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:40.434063 32240 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:15:40.455657 32360 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20260806 20:15:40.447499 30707 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:40.455698 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1006) took 3627 ms. Trace:
I20260806 20:15:40.469544 30866 rpcz_store.cc:276] 0806 20:15:36.828398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:37.253128 (+424730us) service_pool.cc:227] Handling call
0806 20:15:37.441008 (+187880us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:37.441347 (+ 339us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:37.441385 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:37.441502 (+ 117us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:37.441560 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:37.441682 (+ 122us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:40.455522 (+3013840us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54000843d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:40.455637 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":281117440}
I20260806 20:15:40.436759 32089 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.434289 30783 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:40.467339 30756 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:40.442814 32361 raft_consensus.cc:686] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:40.436753 32254 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:40.448736 32286 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:40.446439 32251 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.453794 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:15:40.474850 32365 log.cc:924] Time spent T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260806 20:15:40.446249 32350 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.428218 32297 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:40.475203 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.456863 32295 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.458324 32370 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.485481 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.489645 30784 raft_consensus.cc:2362] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3. Re-sending same reply.
I20260806 20:15:40.490465 32007 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:40.491665 30676 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:40.494365 30756 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:40.466279 31417 raft_consensus.cc:1226] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:40.471874 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.453624 32217 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.466132 32368 consensus_queue.cc:1046] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260806 20:15:40.465857 32369 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 20:15:40.508360 32373 log.cc:924] Time spent T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260806 20:15:40.508837 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.509485 32007 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.529816 32007 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.502015 32103 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:40.507076 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:40.506913 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.519945 30708 raft_consensus.cc:2379] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:40.522925 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.515599 32345 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:40.513969 32270 raft_consensus.cc:2741] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 20:15:40.524142 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.514521 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.529091 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.531790 30676 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.505957 32344 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.534873 30866 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:40.469058 32315 consensus_queue.cc:260] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.527087 32026 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.542285 32244 raft_consensus.cc:3032] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:40.542900 32244 raft_consensus.cc:729] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:40.543923 32026 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.517236 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.545073 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 80) took 2930 ms. Trace:
I20260806 20:15:40.545243 30709 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.545483 30711 rpcz_store.cc:276] 0806 20:15:37.614926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.941289 (+2326363us) service_pool.cc:227] Handling call
0806 20:15:39.991201 (+ 49912us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:39.991982 (+ 781us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:15:40.002294 (+ 10312us) log.cc:842] Serialized 14 byte log entry
0806 20:15:40.125982 (+123688us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:40.143758 (+ 17776us) log.cc:842] Serialized 26 byte log entry
0806 20:15:40.143897 (+ 139us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:40.143931 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:40.144000 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:40.544874 (+400874us) raft_consensus.cc:1671] finished
0806 20:15:40.544897 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:40.545020 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":238,"prepare.queue_time_us":10177,"raft.queue_time_us":3463,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":8653,"spinlock_wait_cycles":3314304,"thread_start_us":97802,"threads_started":1,"wal-append.queue_time_us":188112}
I20260806 20:15:40.548434 30757 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:40.549691 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.551054 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.551203 30783 raft_consensus.cc:2379] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 2: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:40.552183 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:40.552677 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.553252 32013 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.553817 32013 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:40.554442 32013 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.554911 32013 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.556627 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.557808 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.558295 30864 raft_consensus.cc:2379] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 2: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:40.558759 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.559883 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.552789 30711 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:40.538019 32345 raft_consensus.cc:686] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:40.563320 30835 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:40.565884 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.542214 32238 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:40.569715 30676 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.566581 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.567927 30756 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:40.540779 30791 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:40.546865 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.574433 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.577167 30711 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.544945 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:40.577689 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.581099 30707 raft_consensus.cc:2379] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:40.582062 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.581122 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.595373 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:40.577419 32349 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:40.582672 30707 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:40.598826 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:40.575578 32138 consensus_meta.cc:287] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.555s user 0.010s sys 0.002s
I20260806 20:15:40.598434 30708 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:40.573796 32369 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:40.601297 32297 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.602169 32345 consensus_queue.cc:237] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.607139 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.569414 30864 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:40.575820 32026 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:40.610255 31389 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.610733 30784 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:40.590197 32269 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:40.608091 32369 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.583374 32238 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:40.615639 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:40.482826 32361 consensus_queue.cc:237] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.635226 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.633307 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.636629 30866 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:40.631662 30713 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:40.637883 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1030) took 1176 ms. Trace:
I20260806 20:15:40.638553 30713 rpcz_store.cc:276] 0806 20:15:39.461481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.590042 (+128561us) service_pool.cc:227] Handling call
0806 20:15:40.615481 (+1025439us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:40.637827 (+ 22346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2991488}
I20260806 20:15:40.640005 30756 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:40.640071 30757 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:40.544102 32244 consensus_queue.cc:260] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.644577 32297 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:40.645037 32297 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:40.645403 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.647166 30784 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.647233 32293 raft_consensus.cc:3032] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:40.647629 32293 raft_consensus.cc:729] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:40.645417 32297 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:40.648504 32293 consensus_queue.cc:260] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.649943 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.650815 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:40.651298 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.644719 32103 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:40.652308 32103 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.652488 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.653275 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.653936 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.628847 32225 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:40.655351 30835 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:40.652377 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:40.663007 31917 raft_consensus.cc:3032] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:40.664773 31917 raft_consensus.cc:729] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:40.632084 32369 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.629036 32149 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:40.654217 30705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1340) took 1337 ms (client timeout 1621 ms). Trace:
I20260806 20:15:40.661806 30787 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:40.625923 32138 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.629122 32026 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.597684 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:40.656776 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 46604.8us
I20260806 20:15:40.667142 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.659266 32103 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.697252 30866 raft_consensus.cc:2379] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:40.644850 32244 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:40.692576 32274 consensus_meta.cc:287] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.657s user 0.013s sys 0.017s
I20260806 20:15:40.690670 31917 consensus_queue.cc:260] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.702396 31917 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:40.702181 32274 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.691990 32149 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.688741 32293 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.699075 30757 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:40.692584 31386 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.010s sys 0.010s
I20260806 20:15:40.704365 30710 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:40.665535 30864 raft_consensus.cc:1226] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:40.709903 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.693434 32030 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.712338 32274 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:40.693629 32026 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.696543 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:40.667523 32222 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:40.688810 30705 rpcz_store.cc:269] 0806 20:15:39.316854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.319100 (+ 2246us) service_pool.cc:227] Handling call
0806 20:15:40.654045 (+1334945us) spinlock_profiling.cc:243] Waited 9.07 ms on lock 0x7b5400270e54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:40.654151 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20861824}
I20260806 20:15:40.736771 32347 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:40.727413 31386 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.011s sys 0.011s
I20260806 20:15:40.667654 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.785130 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.786934 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:40.741061 30784 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:40.790324 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:40.791013 30866 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:40.792796 30676 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:40.744516 32222 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.799350 30711 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:40.800414 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:40.801075 30711 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:40.803778 30756 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:40.758759 30708 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:40.788458 32149 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:40.809870 30835 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:40.808275 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:40.802240 32352 consensus_meta.cc:287] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.649s user 0.014s sys 0.002s
I20260806 20:15:40.821651 31389 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:40.802062 32222 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:40.828718 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.830999 30756 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:40.831265 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.831913 30711 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:40.826784 30863 consensus_meta.cc:287] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.519s user 0.017s sys 0.000s
I20260806 20:15:40.813658 32246 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:40.839018 30866 raft_consensus.cc:1261] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:40.837024 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:40.819835 32225 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:40.829955 31957 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:40.827075 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:40.827796 32344 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:40.833249 32274 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:40.828737 32322 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:40.852815 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 60116.6us
I20260806 20:15:40.847602 31418 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:40.844689 32010 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:40.870369 30866 raft_consensus.cc:1226] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
W20260806 20:15:40.894130 31388 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.032s sys 0.049s
I20260806 20:15:40.844646 30708 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e"
dest_uuid: "42cabca48db7403b8523a3382a046043"
mode: GRACEFUL
new_leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
from {username='slave'} at 127.0.0.1:38516
W20260806 20:15:40.910221 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 47343.9us
I20260806 20:15:40.839728 30863 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:40.851986 32251 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:40.902771 30864 raft_consensus.cc:1226] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:40.851967 32225 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.848420 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:40.851878 32267 consensus_meta.cc:287] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.673s user 0.014s sys 0.001s
I20260806 20:15:40.843657 32246 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:40.912146 31388 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.629s user 0.032s sys 0.049s
I20260806 20:15:40.856155 32268 consensus_queue.cc:1046] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260806 20:15:40.861698 30705 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:40.861418 32149 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:40.856931 32274 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:40.916148 32280 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:40.861755 32201 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:40.932597 32311 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20260806 20:15:40.864574 31957 raft_consensus.cc:686] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:40.895916 32010 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.830688 32352 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.972029 30785 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:40.861311 32344 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.937361 30708 raft_consensus.cc:595] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Received request to transfer leadership to TS 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:40.861014 32138 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:40.949688 32303 consensus_meta.cc:287] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.591s user 0.007s sys 0.011s
I20260806 20:15:40.975100 30866 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:40.953106 32246 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.113610 30866 raft_consensus.cc:1226] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:40.915021 32007 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.957878 30711 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:41.116184 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.116899 30711 raft_consensus.cc:2354] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3.
I20260806 20:15:40.963074 32149 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:41.117332 32149 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.117723 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.119158 30711 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:41.119663 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1318) took 2906 ms. Trace:
I20260806 20:15:41.119948 30711 rpcz_store.cc:276] 0806 20:15:38.212833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.118725 (+2905892us) service_pool.cc:227] Handling call
0806 20:15:41.119009 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.119459 (+ 450us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.119615 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:40.974049 32274 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:40.976269 32256 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.122372 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:41.123705 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1402) took 2832 ms. Trace:
I20260806 20:15:41.124063 30789 rpcz_store.cc:276] 0806 20:15:38.291659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.326922 (+1035263us) service_pool.cc:227] Handling call
0806 20:15:39.327182 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:39.467906 (+140724us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:39.467943 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:39.467984 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:39.468010 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.123490 (+1655480us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b540026e654. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:41.123542 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:41.123652 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":735,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":105,"spinlock_wait_cycles":322794624}
I20260806 20:15:41.060871 32201 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.963770 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:41.126660 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1196) took 1077 ms. Trace:
I20260806 20:15:41.127072 32201 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.127760 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:40.968727 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:41.081895 32298 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:40.976078 32030 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:41.128554 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1198) took 1078 ms. Trace:
I20260806 20:15:40.981303 32280 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.976159 32103 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.952921 32267 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:40.980819 32219 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.061312 30707 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:41.142292 31388 leader_election.cc:398] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3.
I20260806 20:15:41.142689 31388 leader_election.cc:401] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:15:41.111482 32238 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:41.105041 32341 consensus_meta.cc:287] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.659s user 0.008s sys 0.001s
I20260806 20:15:41.078788 30708 raft_consensus.cc:603] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:41.128916 30789 rpcz_store.cc:276] 0806 20:15:40.050301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.125211 (+1074910us) service_pool.cc:227] Handling call
0806 20:15:41.128475 (+ 3264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:40.976866 32013 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:40.968727 32225 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.090046 30787 raft_consensus.cc:2379] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:41.131162 30866 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:41.130398 32251 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.154065 30789 raft_consensus.cc:1226] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:41.128850 32298 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.161902 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1434) took 1268 ms. Trace:
W20260806 20:15:41.155350 32308 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.032s sys 0.038s
I20260806 20:15:41.116979 32007 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.127022 30784 rpcz_store.cc:276] 0806 20:15:40.049732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.893262 (+843530us) service_pool.cc:227] Handling call
0806 20:15:41.126591 (+233329us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:41.161149 30674 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.014s sys 0.061s
W20260806 20:15:41.195256 30674 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.932s user 0.014s sys 0.061s
I20260806 20:15:41.092916 32303 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.162308 30789 rpcz_store.cc:276] 0806 20:15:39.894018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.151525 (+1257507us) service_pool.cc:227] Handling call
0806 20:15:41.151857 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.161622 (+ 9765us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.161845 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.170329 32346 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.135257 32345 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:41.128947 32030 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:41.158298 30709 consensus_meta.cc:287] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.605s user 0.007s sys 0.009s
W20260806 20:15:41.194242 32308 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s user 0.032s sys 0.039s
I20260806 20:15:41.219277 30865 raft_consensus.cc:1226] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:41.222213 30784 raft_consensus.cc:1226] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:41.222546 32266 consensus_meta.cc:287] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.018s user 0.019s sys 0.000s
W20260806 20:15:41.237146 32275 consensus_meta.cc:287] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.807s user 0.006s sys 0.013s
I20260806 20:15:41.225270 31423 raft_consensus.cc:2396] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:41.226706 32267 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:41.232318 32297 consensus_meta.cc:287] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.583s user 0.006s sys 0.003s
I20260806 20:15:41.300549 32322 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.300416 32297 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:40.645393 32257 consensus_queue.cc:1159] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" responder_term: 2 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260806 20:15:41.301288 30788 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53732 (request call id 61) took 1465 ms (client timeout 1775 ms). Trace:
W20260806 20:15:41.232875 31917 consensus_meta.cc:287] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.530s user 0.008s sys 0.004s
I20260806 20:15:41.301083 32322 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:41.301723 30788 rpcz_store.cc:269] 0806 20:15:39.836013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.864956 (+ 28943us) service_pool.cc:227] Handling call
0806 20:15:41.301129 (+1436173us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b54000131d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:41.301233 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55216512}
I20260806 20:15:41.303035 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:41.303645 30788 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:41.304124 30789 raft_consensus.cc:1226] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:41.303839 32297 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:41.305954 32074 raft_consensus.cc:3032] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:41.306350 32074 raft_consensus.cc:729] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:41.237110 32026 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.307104 32074 consensus_queue.cc:260] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.308079 32074 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.308389 30783 raft_consensus.cc:2354] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3.
I20260806 20:15:41.158710 32317 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
W20260806 20:15:41.235123 32296 consensus_meta.cc:287] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.842s user 0.011s sys 0.005s
I20260806 20:15:41.262109 30790 raft_consensus.cc:1226] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 3. Ops: []
I20260806 20:15:41.246089 30787 raft_consensus.cc:1226] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
W20260806 20:15:41.251418 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1017) took 1790 ms (client timeout 2193 ms). Trace:
I20260806 20:15:41.218256 32352 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.250344 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1021) took 1778 ms. Trace:
W20260806 20:15:41.229244 32142 consensus_meta.cc:287] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.533s user 0.002s sys 0.004s
W20260806 20:15:41.222386 32379 log.cc:924] Time spent T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260806 20:15:41.278597 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1029) took 1660 ms. Trace:
I20260806 20:15:41.194005 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:38516 (request call id 4) took 1008 ms. Trace:
W20260806 20:15:41.243095 32293 consensus_meta.cc:287] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.506s user 0.009s sys 0.000s
I20260806 20:15:41.199128 32361 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:41.248420 30863 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
W20260806 20:15:41.314900 32370 consensus_meta.cc:287] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.804s user 0.009s sys 0.005s
I20260806 20:15:41.248063 32303 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.263787 32013 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:41.294888 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 351330us
I20260806 20:15:41.327136 31389 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:41.160955 31957 consensus_queue.cc:237] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.262468 32280 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:41.234822 32339 consensus_meta.cc:287] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.928s user 0.008s sys 0.007s
I20260806 20:15:41.317921 30708 rpcz_store.cc:276] 0806 20:15:40.153729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.835311 (+681582us) service_pool.cc:227] Handling call
0806 20:15:41.159477 (+324166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.336879 32339 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:41.337822 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.339172 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.339860 30708 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:41.340777 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.295658 32219 consensus_queue.cc:1046] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260806 20:15:41.191540 32251 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.304560 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:41.310330 30835 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:41.312373 30869 rpcz_store.cc:269] 0806 20:15:39.461346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.462499 (+ 1153us) service_pool.cc:227] Handling call
0806 20:15:41.251264 (+1788765us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b5400241dd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:41.251365 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":451287936}
I20260806 20:15:41.347584 30788 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:41.312865 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1188) took 1279 ms. Trace:
I20260806 20:15:41.313360 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1187) took 1280 ms. Trace:
I20260806 20:15:41.350766 30787 rpcz_store.cc:276] 0806 20:15:40.032973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.245556 (+1212583us) service_pool.cc:227] Handling call
0806 20:15:41.245890 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.313166 (+ 67276us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.313318 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.312924 30871 rpcz_store.cc:276] 0806 20:15:39.471997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.472195 (+ 198us) service_pool.cc:227] Handling call
0806 20:15:39.588199 (+116004us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:40.274932 (+686733us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.250114 (+975182us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b540007e254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:41.250288 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1552527104}
I20260806 20:15:41.257144 32337 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.259291 32330 raft_consensus.cc:3032] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:41.308071 32026 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.277417 30709 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:41.369102 30834 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.295271 32266 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:41.314916 30756 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:41.342121 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:41.293885 32007 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:41.319582 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 368424us
I20260806 20:15:41.225108 32246 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.245903 32149 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.278369 32345 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.314633 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1186) took 1281 ms. Trace:
I20260806 20:15:41.157276 32341 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.245957 32010 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:41.323845 32347 consensus_meta.cc:287] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.535s user 0.011s sys 0.001s
I20260806 20:15:41.317332 32346 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3.
I20260806 20:15:41.329404 32030 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:41.334028 30705 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:41.331120 32271 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:41.334695 30703 raft_consensus.cc:2396] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:41.351745 30790 rpcz_store.cc:276] 0806 20:15:40.033249 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.261568 (+1228319us) service_pool.cc:227] Handling call
0806 20:15:41.261917 (+ 349us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.312630 (+ 50713us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.312810 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.316526 32296 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.355592 30870 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:41.341338 30708 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.316812 30865 rpcz_store.cc:276] 0806 20:15:39.618105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.218653 (+1600548us) service_pool.cc:227] Handling call
0806 20:15:41.219084 (+ 431us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.278327 (+ 59243us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.278539 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.390525 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:41.295473 32275 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.391920 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.262742 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:41.392577 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 77746.1us
W20260806 20:15:41.486938 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 95084.5us
I20260806 20:15:41.487708 30834 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.342916 32344 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.353360 32280 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:41.392591 32339 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:41.448863 32007 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.506152 32007 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.507138 32341 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.493394 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:41.509279 31414 raft_consensus.cc:2354] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:41.447064 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:41.529053 30871 raft_consensus.cc:1226] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:41.324852 32361 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.353547 32195 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:41.532883 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260806 20:15:41.462584 30788 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:41.524314 32282 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:41.391816 32292 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:41.353745 32103 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.391969 32288 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:41.392534 32330 raft_consensus.cc:729] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:41.445202 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1350) took 1220 ms. Trace:
I20260806 20:15:41.450829 32238 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.457589 32368 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:41.546692 32368 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.547210 32010 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.512491 30783 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 3. Ops: []
I20260806 20:15:41.548375 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1189) took 1514 ms. Trace:
I20260806 20:15:41.548914 32368 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.549136 32361 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.542702 30709 rpcz_store.cc:276] 0806 20:15:40.175921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.483838 (+307917us) service_pool.cc:227] Handling call
0806 20:15:41.395861 (+912023us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.550539 31418 raft_consensus.cc:1226] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:41.550669 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.551450 30709 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 5
I20260806 20:15:41.551900 32343 raft_consensus.cc:3032] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:41.552310 32343 raft_consensus.cc:729] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:41.465613 32235 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:41.357327 32013 consensus_queue.cc:1046] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.114s
I20260806 20:15:41.557209 32013 raft_consensus.cc:3032] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:41.557574 32013 raft_consensus.cc:729] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:41.354422 32267 consensus_queue.cc:1046] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.119s
I20260806 20:15:41.487840 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1023) took 4008 ms. Trace:
I20260806 20:15:41.558234 30710 rpcz_store.cc:276] 0806 20:15:37.479298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.870140 (+2390842us) service_pool.cc:227] Handling call
0806 20:15:39.901026 (+ 30886us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:40.727909 (+826883us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.487608 (+759699us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b5400270954. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:41.487787 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1847066368}
I20260806 20:15:41.559542 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.560231 30710 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:41.560992 32298 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:41.358633 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 48395.4us
I20260806 20:15:41.488869 30834 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.563614 32275 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:41.563849 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 75047.6us
I20260806 20:15:41.564235 32301 raft_consensus.cc:3032] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:41.564608 32301 raft_consensus.cc:729] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:15:41.478507 32222 consensus_meta.cc:287] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.641s user 0.007s sys 0.003s
I20260806 20:15:41.457975 32352 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260806 20:15:41.496917 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:41.472481 32347 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:41.565358 32222 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.500379 32280 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.566567 30790 raft_consensus.cc:2454] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:41.504285 32296 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.510063 32345 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.510351 32149 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:41.511121 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.512446 32251 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.513630 30868 raft_consensus.cc:2396] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:41.514212 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 56) took 4938 ms. Trace:
I20260806 20:15:41.516713 32367 raft_consensus.cc:3032] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:41.517730 32246 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.494118 32231 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:41.516746 30861 raft_consensus.cc:2379] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:41.493718 32271 raft_consensus.cc:686] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:41.463371 30784 rpcz_store.cc:276] 0806 20:15:40.032718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.221566 (+1188848us) service_pool.cc:227] Handling call
0806 20:15:41.222000 (+ 434us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.290089 (+ 68089us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.312779 (+ 22690us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.498653 30707 raft_consensus.cc:1226] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:41.472056 32391 consensus_queue.cc:1046] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260806 20:15:41.449800 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:41.567632 32282 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:41.548709 30783 rpcz_store.cc:276] 0806 20:15:40.038676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.490299 (+1451623us) service_pool.cc:227] Handling call
0806 20:15:41.512297 (+ 21998us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.548094 (+ 35797us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.548315 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.524546 32250 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.479781 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:41.527905 30713 raft_consensus.cc:2379] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:41.592931 32367 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:41.528975 30866 raft_consensus.cc:1226] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:41.542651 32195 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.535378 31389 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:41.533370 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:41.538255 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.540472 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:15:41.609062 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 216218us
W20260806 20:15:41.608863 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 73597.1us
I20260806 20:15:41.609709 30788 raft_consensus.cc:2379] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:15:41.483014 32268 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.017s sys 0.034s
I20260806 20:15:41.544023 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:15:41.542156 32288 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.532778 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:41.545521 32238 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.554075 31422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 263) took 3262 ms. Trace:
I20260806 20:15:41.556442 32235 raft_consensus.cc:686] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:41.567458 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:41.619823 30790 raft_consensus.cc:2379] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:41.569125 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:41.621143 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:41.621519 30754 leader_election.cc:398] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:41.621773 30871 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.570828 30834 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:41.622639 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 51883.3us
I20260806 20:15:41.561304 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.623363 30834 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:41.623751 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.624891 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.625248 32288 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.536175 32292 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:41.625828 32292 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.606482 32274 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.626843 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:41.627588 32274 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.603538 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 76) took 1415 ms. Trace:
I20260806 20:15:41.628710 30861 rpcz_store.cc:276] 0806 20:15:40.186782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.192216 (+ 5434us) service_pool.cc:227] Handling call
0806 20:15:41.262563 (+1070347us) spinlock_profiling.cc:243] Waited 7.02 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:41.601683 (+339120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16142720}
I20260806 20:15:41.593900 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1320) took 3363 ms. Trace:
I20260806 20:15:41.567920 32103 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.590597 30705 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.523463 32315 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
W20260806 20:15:41.523845 32369 consensus_meta.cc:287] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.825s user 0.007s sys 0.003s
I20260806 20:15:41.597651 32391 consensus_queue.cc:1225] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer 58d59beb597f4ed99d84d102ddb41ff5 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 20:15:41.617429 32267 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:41.570715 32280 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:41.618877 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:41.686695 30705 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:41.591153 30785 rpcz_store.cc:276] 0806 20:15:36.557799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.576792 (+ 18993us) service_pool.cc:227] Handling call
0806 20:15:36.669539 (+ 92747us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.068667 (+4399128us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.479360 (+410693us) spinlock_profiling.cc:243] Waited 4.3 s on lock 0x7b54003d45d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:41.494675 (+ 15315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9887848320}
I20260806 20:15:41.687319 32267 raft_consensus.cc:686] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:41.687855 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:41.688560 30785 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:41.688328 32267 consensus_queue.cc:237] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.605324 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:41.666153 32103 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.690512 30791 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.690603 30784 raft_consensus.cc:1226] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:41.691421 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1195) took 1642 ms. Trace:
I20260806 20:15:41.605822 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:41.691569 32103 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.691996 30787 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.604641 31416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 273) took 1667 ms. Trace:
I20260806 20:15:41.611155 31389 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:41.637424 32282 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:41.615876 31422 rpcz_store.cc:276] 0806 20:15:38.291463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.291580 (+ 117us) service_pool.cc:227] Handling call
0806 20:15:39.800038 (+1508458us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:39.800270 (+ 232us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:39.810950 (+ 10680us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:39.810976 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:39.821847 (+ 10871us) log.cc:842] Serialized 26 byte log entry
0806 20:15:41.128090 (+1306243us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b18002ae5c0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e735345 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb 00007f71512f6063 00007f71512f6411 00007f71516435cf
0806 20:15:41.352416 (+224326us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:41.352451 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.352522 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:41.553835 (+201313us) raft_consensus.cc:1671] finished
0806 20:15:41.553859 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:41.554012 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3384,"raft.queue_time_us":10386,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":13877,"spinlock_wait_cycles":5585408,"thread_start_us":224159,"threads_started":1,"wal-append.queue_time_us":1655176}
I20260806 20:15:41.692989 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:41.621861 30754 leader_election.cc:401] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 20:15:41.630581 30834 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.630976 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:41.704078 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 73574us
I20260806 20:15:41.631382 30710 raft_consensus.cc:2379] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:41.703222 30787 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:41.633028 30867 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:41.705829 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 78) took 1518 ms. Trace:
I20260806 20:15:41.630115 30711 rpcz_store.cc:276] 0806 20:15:38.230673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.120686 (+2890013us) service_pool.cc:227] Handling call
0806 20:15:41.121000 (+ 314us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:41.121314 (+ 314us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:41.121341 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:41.134192 (+ 12851us) log.cc:842] Serialized 26 byte log entry
0806 20:15:41.471904 (+337712us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:41.471947 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.472022 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:41.593617 (+121595us) raft_consensus.cc:1671] finished
0806 20:15:41.593640 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:41.593828 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":195,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":130,"thread_start_us":311791,"threads_started":1,"wal-append.queue_time_us":349723}
I20260806 20:15:41.634832 30871 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:41.558338 32013 consensus_queue.cc:260] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.629611 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:41.708997 30866 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.701678 32361 raft_consensus.cc:2781] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:41.709647 32361 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:41.709766 32013 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.710124 32361 raft_consensus.cc:3037] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:41.710502 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.711093 30711 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:41.614069 32268 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.645s user 0.018s sys 0.035s
I20260806 20:15:41.608098 32325 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:49046 (request call id 378) took 5181 ms. Trace:
I20260806 20:15:41.713423 32027 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.715561 30866 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:41.714620 32325 rpcz_store.cc:276] 0806 20:15:36.421505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.421893 (+ 388us) service_pool.cc:227] Handling call
0806 20:15:37.989552 (+1567659us) spinlock_profiling.cc:243] Waited 3.93 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f714e7355fe 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71565fbda2
0806 20:15:38.925709 (+936157us) ts_tablet_manager.cc:928] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
0806 20:15:40.163105 (+1237396us) tablet_copy_client.cc:361] Tablet copy session begun
0806 20:15:41.257434 (+1094329us) tablet_metadata.cc:641] Metadata flushed
0806 20:15:41.257800 (+ 366us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 20:15:41.527608 (+269808us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 20:15:41.528000 (+ 392us) tablet_replica.cc:205] Creating consensus instance
0806 20:15:41.567717 (+ 39717us) ts_tablet_manager.cc:983] Replica registered.
0806 20:15:41.602566 (+ 34849us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":155838,"spinlock_wait_cycles":9035648,"tablet-copy.queue_time_us":2444223,"thread_start_us":2526013,"threads_started":2}
I20260806 20:15:41.716230 32325 tablet_copy_client.cc:791] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:41.565387 32301 consensus_queue.cc:260] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.567462 32222 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.666527 31420 raft_consensus.cc:2379] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:41.666462 32347 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.717414 32301 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:41.717717 32347 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.718356 30708 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:41.667042 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:15:41.666391 32369 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.668088 30865 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:41.667565 32367 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.723953 32367 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.604862 32271 consensus_queue.cc:237] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.669744 30864 raft_consensus.cc:2379] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:41.669201 32235 consensus_queue.cc:237] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.636667 32231 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:41.615388 32368 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:41.573997 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:41.682593 32238 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:41.672906 32292 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.726306 32374 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.548817 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:41.693724 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 82660.7us
W20260806 20:15:41.613631 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 75487.7us
I20260806 20:15:41.695637 30785 raft_consensus.cc:1203] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20260806 20:15:41.695915 32246 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:41.698190 30756 leader_election.cc:334] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Timed out: RequestConsensusVote RPC to 127.28.186.193:42937 timed out after 1.638s (SENT)
I20260806 20:15:41.697624 32282 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:41.695433 31416 rpcz_store.cc:276] 0806 20:15:39.937543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.942719 (+ 5176us) service_pool.cc:227] Handling call
0806 20:15:41.244105 (+1301386us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71bbce 000000000034aea9 00007f715653ba30 00007f71566017b4 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:41.244281 (+ 176us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.533509 (+289228us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:41.533539 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:41.539126 (+ 5587us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:41.539186 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.596338 (+ 57152us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b5400661554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:41.598258 (+ 1920us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:41.603907 (+ 5649us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":670955136}
I20260806 20:15:41.553381 32142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:49038 (request call id 218) took 10626 ms. Trace:
I20260806 20:15:41.706547 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.706139 30867 rpcz_store.cc:276] 0806 20:15:40.187360 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.212315 (+ 24955us) service_pool.cc:227] Handling call
0806 20:15:41.390355 (+1178040us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:41.705752 (+315397us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12478336}
I20260806 20:15:41.707711 30787 raft_consensus.cc:1226] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
W20260806 20:15:41.757404 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 50920us
I20260806 20:15:41.741616 32347 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.705055 32283 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:41.708328 32251 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:41.751811 32368 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.714862 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.764130 30710 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.756642 32282 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:41.722069 30711 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:41.731339 32369 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.754961 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:41.716766 32325 tablet_copy_client.cc:655] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:41.740442 32231 raft_consensus.cc:686] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:41.740657 31417 raft_consensus.cc:2396] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:41.686633 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:41.742800 30791 raft_consensus.cc:2396] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:41.727360 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.746599 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1185) took 1714 ms. Trace:
I20260806 20:15:41.760357 32283 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:41.756510 32246 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.750860 30676 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.755614 30869 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:41.756102 30756 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:41.707640 30835 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:41.758970 30834 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.781487 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.782244 30834 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.782553 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.759788 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1438) took 1590 ms. Trace:
I20260806 20:15:41.713717 32275 consensus_queue.cc:1046] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
W20260806 20:15:41.711717 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 92150.3us
W20260806 20:15:41.755139 31415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49046 (request call id 403) took 1555 ms (client timeout 1731 ms). Trace:
I20260806 20:15:41.758042 30785 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:41.766840 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.737210 32285 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.769569 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:41.795475 31422 raft_consensus.cc:2354] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4.
I20260806 20:15:41.770686 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1051) took 1735 ms. Trace:
I20260806 20:15:41.796424 30713 rpcz_store.cc:276] 0806 20:15:40.035101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.652059 (+616958us) service_pool.cc:227] Handling call
0806 20:15:41.770475 (+1118416us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b54000d5f54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:41.770625 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1862257024}
I20260806 20:15:41.797848 30832 leader_election.cc:398] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4.
I20260806 20:15:41.798172 30832 leader_election.cc:401] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 20:15:41.771906 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:41.759625 32222 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:41.758766 30709 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:41.747380 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 77) took 1560 ms. Trace:
I20260806 20:15:41.753671 32010 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260806 20:15:41.775367 30789 rpcz_store.cc:276] 0806 20:15:40.032442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.201147 (+1168705us) service_pool.cc:227] Handling call
0806 20:15:41.201503 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.304484 (+102981us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.746361 (+441877us) spinlock_profiling.cc:243] Waited 88 ms on lock 0x7b54003b24d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:41.746542 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":202309376}
I20260806 20:15:41.799806 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1439) took 1537 ms. Trace:
I20260806 20:15:41.802973 30788 rpcz_store.cc:276] 0806 20:15:40.262286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.799032 (+1536746us) service_pool.cc:227] Handling call
0806 20:15:41.799346 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.799514 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:41.799538 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:41.799578 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:41.799605 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.799652 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:41.799753 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.804347 30788 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:41.804924 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1442) took 1514 ms. Trace:
I20260806 20:15:41.779744 32283 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.745162 32330 consensus_queue.cc:260] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.761914 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:41.786691 32277 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:41.794710 31957 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:41.809769 30708 raft_consensus.cc:2354] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:41.785504 30676 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.765851 32251 raft_consensus.cc:686] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:41.800325 30870 rpcz_store.cc:276] 0806 20:15:40.187074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.189686 (+ 2612us) service_pool.cc:227] Handling call
0806 20:15:41.747225 (+1557539us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400375d54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:41.747327 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2375150336}
I20260806 20:15:41.801573 30673 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.752007 32374 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:41.779495 32232 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.776955 32396 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.780007 30783 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.816776 32242 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:41.805194 30788 rpcz_store.cc:276] 0806 20:15:40.290299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.803912 (+1513613us) service_pool.cc:227] Handling call
0806 20:15:41.804172 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.804711 (+ 539us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.804877 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:41.809718 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 111567us
I20260806 20:15:41.809896 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1440) took 1542 ms. Trace:
I20260806 20:15:41.809718 30710 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
W20260806 20:15:41.781153 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 73591.8us
I20260806 20:15:41.766604 32368 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:41.818480 31957 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:41.790772 31415 rpcz_store.cc:269] 0806 20:15:40.199575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.201704 (+ 2129us) service_pool.cc:227] Handling call
0806 20:15:41.548696 (+1346992us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71bbce 000000000034aea9 00007f715653ba30 00007f71566017b4 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:41.753209 (+204513us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5221632}
I20260806 20:15:41.791941 32238 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:41.794364 32369 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:41.785161 30834 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.814541 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.817576 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.794515 32337 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:41.783418 30787 rpcz_store.cc:276] 0806 20:15:40.169720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.707272 (+1537552us) service_pool.cc:227] Handling call
0806 20:15:41.707553 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.759530 (+ 51977us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.759732 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.794898 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:41.814455 32222 raft_consensus.cc:686] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:41.757891 32142 rpcz_store.cc:276] 0806 20:15:30.943673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:30.991433 (+ 47760us) service_pool.cc:227] Handling call
0806 20:15:33.879736 (+2888303us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71be82 00007f71576288b9 00007f715653b5d8 00007f71565fc123 00007f715662da8e 00007f71566135ad 00007f7156613559 00007f71566134e9 00007f71566134b1
0806 20:15:34.340622 (+460886us) ts_tablet_manager.cc:928] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
0806 20:15:40.196138 (+5855516us) tablet_copy_client.cc:361] Tablet copy session begun
0806 20:15:40.597595 (+401457us) tablet_metadata.cc:641] Metadata flushed
0806 20:15:40.660766 (+ 63171us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 20:15:41.325318 (+664552us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 20:15:41.393576 (+ 68258us) tablet_replica.cc:205] Creating consensus instance
0806 20:15:41.553113 (+159537us) ts_tablet_manager.cc:983] Replica registered.
0806 20:15:41.553292 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":297,"spinlock_wait_cycles":756436224,"tablet-copy.queue_time_us":1123321,"thread_start_us":2054992,"threads_started":2}
I20260806 20:15:41.865619 32142 tablet_copy_client.cc:791] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 0 data blocks...
I20260806 20:15:41.865419 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:41.866192 30787 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.866250 32142 tablet_copy_client.cc:655] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Starting download of 1 WAL segments...
I20260806 20:15:41.868454 32246 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.869164 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.822939 32027 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:41.825117 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.825644 30786 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:41.826267 32292 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.827018 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1190) took 1788 ms. Trace:
W20260806 20:15:41.806133 30703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1307) took 3944 ms (client timeout 3195 ms). Trace:
I20260806 20:15:41.879725 32234 consensus_queue.cc:1046] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:15:41.880004 30703 rpcz_store.cc:269] 0806 20:15:37.851303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.523763 (+2672460us) service_pool.cc:227] Handling call
0806 20:15:41.778525 (+1254762us) spinlock_profiling.cc:243] Waited 786 ms on lock 0x7b54001b4754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:41.795045 (+ 16520us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1807022464}
W20260806 20:15:41.832486 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 47082.2us
I20260806 20:15:41.815050 32268 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:41.880589 32234 raft_consensus.cc:3032] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:41.880939 32234 raft_consensus.cc:729] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:41.837201 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1443) took 1537 ms. Trace:
I20260806 20:15:41.854131 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.837270 30756 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:41.871517 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:41.815207 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 47673.1us
I20260806 20:15:41.856127 30706 raft_consensus.cc:2396] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:41.786967 32257 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:41.904006 32257 raft_consensus.cc:686] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:41.904902 32257 consensus_queue.cc:237] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.906603 32374 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.862169 30713 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:41.907846 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.908959 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.835386 32247 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:41.834839 32277 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:41.864620 32251 consensus_queue.cc:237] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.898195 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1043) took 1366 ms. Trace:
W20260806 20:15:41.869494 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 84400.8us
I20260806 20:15:41.912367 30863 rpcz_store.cc:276] 0806 20:15:40.531742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.962030 (+430288us) service_pool.cc:227] Handling call
0806 20:15:40.974502 (+ 12472us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.324356 (+349854us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.897932 (+573576us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54000843d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:41.898130 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":623314688}
I20260806 20:15:41.869946 30673 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:41.913857 30870 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:41.858839 31957 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:41.914260 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 44412.8us
I20260806 20:15:41.915726 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:41.916463 30865 raft_consensus.cc:2379] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:41.917156 32277 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:41.917798 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:41.918017 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:41.918609 31388 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.919263 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:41.920989 31389 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:41.834537 32232 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.922097 31389 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:41.894791 32368 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:41.924034 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:41.923836 31957 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.925858 32195 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.926560 32195 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:41.927042 32195 raft_consensus.cc:686] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:41.927385 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:41.874037 30832 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:41.927982 30870 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:41.928673 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 54707us
I20260806 20:15:41.825763 32231 consensus_queue.cc:237] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.931108 32330 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.692951 30784 rpcz_store.cc:276] 0806 20:15:40.049464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.690062 (+1640598us) service_pool.cc:227] Handling call
0806 20:15:41.690419 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.691164 (+ 745us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.691367 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.884121 30676 leader_election.cc:398] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3.
I20260806 20:15:41.954391 30676 leader_election.cc:401] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260806 20:15:41.955106 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 71128.8us
I20260806 20:15:41.886432 30710 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:41.872330 32339 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 20:15:41.886776 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 49636.1us
I20260806 20:15:41.959283 30756 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:41.960201 30756 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:41.852962 30789 rpcz_store.cc:276] 0806 20:15:40.267728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.803034 (+1535306us) service_pool.cc:227] Handling call
0806 20:15:41.803299 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.809591 (+ 6292us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:41.809616 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:41.809651 (+ 35us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:41.809675 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.809736 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:41.809846 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.886180 30788 rpcz_store.cc:276] 0806 20:15:40.299988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.836493 (+1536505us) service_pool.cc:227] Handling call
0806 20:15:41.836764 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.836915 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:41.836937 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:41.836973 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:41.836998 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.837045 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:41.837132 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.877740 30785 rpcz_store.cc:276] 0806 20:15:40.038990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.689273 (+1650283us) service_pool.cc:227] Handling call
0806 20:15:41.695524 (+ 6251us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:41.809980 (+114456us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:41.826954 (+ 16974us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.887667 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.889516 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.874614 31417 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:41.894021 32222 consensus_queue.cc:237] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:41.881767 32234 consensus_queue.cc:260] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.887639 32347 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:41.875648 32369 consensus_queue.cc:1046] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260806 20:15:41.900051 32322 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.899845 32361 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:41.900888 31423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 285) took 2078 ms (client timeout 2206 ms). Trace:
I20260806 20:15:41.973567 32368 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3.
I20260806 20:15:41.902493 32078 raft_consensus.cc:3032] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:41.903431 32311 raft_consensus.cc:3032] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:41.911495 32343 consensus_queue.cc:260] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.007230 32343 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:42.007485 32361 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.997027 32339 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:42.008591 32282 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.927199 32266 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:42.009747 32266 raft_consensus.cc:686] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:42.010528 32227 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:42.010694 32266 consensus_queue.cc:237] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:42.003017 32078 raft_consensus.cc:729] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:41.927055 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:41.871663 30711 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:41.933501 30713 raft_consensus.cc:2379] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:41.935683 30870 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:41.994292 30705 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:41.934595 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:41.944576 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:42.019234 30866 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:41.946000 32247 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.946214 32142 tablet_copy_client.cc:531] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:41.946990 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:41.947732 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:41.947916 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:41.946224 30708 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:41.945899 32244 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:41.948956 32378 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:41.949958 31957 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.951184 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:41.928808 30709 raft_consensus.cc:2396] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 20:15:41.962023 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:42.033052 32378 raft_consensus.cc:686] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:42.008586 32339 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.033987 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:42.034621 30789 raft_consensus.cc:3037] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:42.034940 32339 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:41.981631 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:42.036384 30788 raft_consensus.cc:1226] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:42.036931 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1447) took 1599 ms. Trace:
I20260806 20:15:41.964275 30756 leader_election.cc:398] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3.
I20260806 20:15:42.037228 30788 rpcz_store.cc:276] 0806 20:15:40.437600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.035932 (+1598332us) service_pool.cc:227] Handling call
0806 20:15:42.036224 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:42.036721 (+ 497us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:42.036877 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:41.995783 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:41.881404 32402 log.cc:924] Time spent T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.142s user 0.001s sys 0.001s
I20260806 20:15:42.038491 30788 raft_consensus.cc:3032] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
W20260806 20:15:42.000368 31423 rpcz_store.cc:269] 0806 20:15:39.822745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:39.837387 (+ 14642us) service_pool.cc:227] Handling call
0806 20:15:41.900687 (+2063300us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54005ed8d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:41.900829 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3178465664}
I20260806 20:15:42.002636 30676 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:42.038825 30788 raft_consensus.cc:729] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:41.988204 32331 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:42.039235 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:42.039534 32331 raft_consensus.cc:686] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:42.039620 30788 consensus_queue.cc:260] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:42.041028 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 76925.4us
I20260806 20:15:41.993664 30703 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:42.002013 32288 consensus_queue.cc:1046] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260806 20:15:41.982831 32281 consensus_queue.cc:1046] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260806 20:15:41.927959 32195 consensus_queue.cc:237] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:42.042896 32047 raft_consensus.cc:3032] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:42.043473 32047 raft_consensus.cc:729] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:42.043889 30710 raft_consensus.cc:2379] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:42.044746 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:41.835525 32396 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:42.045320 32393 consensus_queue.cc:1046] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:42.045797 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:41.953641 32013 consensus_queue.cc:1046] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260806 20:15:42.047250 32374 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:42.047866 32013 raft_consensus.cc:3032] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:42.048307 32013 raft_consensus.cc:729] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:41.876004 30787 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:42.049067 32013 consensus_queue.cc:260] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:41.999181 32291 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:42.050261 32291 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.050710 30787 raft_consensus.cc:1261] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:42.051280 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1207) took 1395 ms. Trace:
I20260806 20:15:42.051573 30787 rpcz_store.cc:276] 0806 20:15:40.655870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.050093 (+1394223us) service_pool.cc:227] Handling call
0806 20:15:42.050395 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:42.051061 (+ 666us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:42.051231 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.041779 30785 raft_consensus.cc:1226] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:42.052264 32291 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:42.052618 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:42.017545 30867 raft_consensus.cc:2379] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:41.911530 32283 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.020259 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:42.049592 30791 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1435) took 2052 ms (client timeout 2147 ms). Trace:
I20260806 20:15:42.054152 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:42.054657 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:42.055011 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1323) took 3791 ms. Trace:
I20260806 20:15:42.019425 30871 raft_consensus.cc:2379] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:42.022485 30865 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:42.055279 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:42.055907 30867 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:42.056900 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:41.983927 32223 consensus_queue.cc:1046] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260806 20:15:42.057533 30707 rpcz_store.cc:276] 0806 20:15:38.263311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.159788 (+2896477us) service_pool.cc:227] Handling call
0806 20:15:41.479389 (+319601us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:41.598080 (+118691us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:42.054819 (+456739us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b5400211bd0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:42.054953 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":555640576}
I20260806 20:15:42.058707 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:42.059438 30707 raft_consensus.cc:2396] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 4 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:42.060170 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:42.060894 32283 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:42.061503 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:42.062111 30707 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:42.062826 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:42.024113 32325 tablet_copy_client.cc:531] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:15:42.063591 30710 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:42.063922 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:42.028718 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:42.048180 32288 raft_consensus.cc:3032] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:42.003314 32347 raft_consensus.cc:686] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:42.024474 32247 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:42.046684 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 44208.4us
I20260806 20:15:42.021930 32268 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260806 20:15:42.034051 32378 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:42.023027 31415 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:41.999909 32149 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:41.981365 32337 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:42.254133 32149 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.254721 32337 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:42.256233 32149 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:42.256589 32337 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:41.995599 32370 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:42.257264 32370 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.040796 32331 consensus_queue.cc:237] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.259447 32370 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:41.994438 32388 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:42.262228 32388 raft_consensus.cc:686] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:42.067415 32013 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:42.017848 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:42.062964 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:42.941347 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1373) took 1020 ms. Trace:
I20260806 20:15:42.941744 30707 rpcz_store.cc:276] 0806 20:15:41.920900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.062639 (+141739us) service_pool.cc:227] Handling call
0806 20:15:42.941280 (+878641us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.943147 30707 raft_consensus.cc:1226] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:42.943662 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1324) took 4641 ms. Trace:
I20260806 20:15:42.943862 30707 rpcz_store.cc:276] 0806 20:15:38.302400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.942597 (+4640197us) service_pool.cc:227] Handling call
0806 20:15:42.942970 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:42.943485 (+ 515us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:42.943626 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.041828 30756 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:42.947362 32421 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:42.947814 32421 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:42.949054 32378 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.014s sys 0.005s
W20260806 20:15:42.949433 32378 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s user 0.015s sys 0.005s
I20260806 20:15:42.949842 32421 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:42.004941 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:42.951931 30784 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:42.052794 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1444) took 1745 ms. Trace:
I20260806 20:15:42.974947 30785 rpcz_store.cc:276] 0806 20:15:40.307085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.994104 (+1687019us) service_pool.cc:227] Handling call
0806 20:15:42.025198 (+ 31094us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:42.052527 (+ 27329us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:42.052735 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.052835 30787 raft_consensus.cc:1261] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:15:42.990931 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1454) took 2276 ms. Trace:
I20260806 20:15:42.991299 30787 rpcz_store.cc:276] 0806 20:15:40.714331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.052234 (+1337903us) service_pool.cc:227] Handling call
0806 20:15:42.052532 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:42.990608 (+938076us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:42.990866 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.992404 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:42.040624 30788 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:42.064437 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:42.086563 30835 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:43.002715 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 916173us
I20260806 20:15:42.004241 30834 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:42.055912 30706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1049) took 2021 ms (client timeout 1638 ms). Trace:
W20260806 20:15:43.010535 30706 rpcz_store.cc:269] 0806 20:15:40.034571 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.133014 (+ 98443us) service_pool.cc:227] Handling call
0806 20:15:42.055723 (+1922709us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54000d6454. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:42.055853 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3889371904}
I20260806 20:15:43.011967 30706 raft_consensus.cc:1226] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 1. Current term is 3. Ops: []
I20260806 20:15:43.012557 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1327) took 4677 ms. Trace:
I20260806 20:15:43.012871 30706 rpcz_store.cc:276] 0806 20:15:38.335108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.011416 (+4676308us) service_pool.cc:227] Handling call
0806 20:15:43.011802 (+ 386us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.012321 (+ 519us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.012511 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.089252 30861 raft_consensus.cc:2379] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:43.020123 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1237) took 1029 ms. Trace:
I20260806 20:15:43.020483 30861 rpcz_store.cc:276] 0806 20:15:41.990378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.052344 (+ 61966us) service_pool.cc:227] Handling call
0806 20:15:43.020046 (+967702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.031780 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:42.072793 30865 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:43.023664 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1055) took 1097 ms. Trace:
I20260806 20:15:43.023998 30865 rpcz_store.cc:276] 0806 20:15:41.926629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.931636 (+ 5007us) service_pool.cc:227] Handling call
0806 20:15:43.023577 (+1091941us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.025123 30865 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:42.062744 30864 raft_consensus.cc:1226] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:15:43.028360 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1052) took 1102 ms. Trace:
I20260806 20:15:43.028723 30864 rpcz_store.cc:276] 0806 20:15:41.925585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.927457 (+ 1872us) service_pool.cc:227] Handling call
0806 20:15:41.948963 (+ 21506us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:43.026402 (+1077439us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.028074 (+ 1672us) spinlock_profiling.cc:243] Waited 75.4 ms on lock 0x7b5400240ed4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:43.028296 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":173307136}
I20260806 20:15:43.029613 32369 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.030170 32369 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.042914 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.032136 30869 raft_consensus.cc:2379] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:43.032622 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1235) took 1043 ms. Trace:
I20260806 20:15:43.032928 30869 rpcz_store.cc:276] 0806 20:15:41.989590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.042656 (+ 53066us) service_pool.cc:227] Handling call
0806 20:15:43.032551 (+989895us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.069272 30867 raft_consensus.cc:2379] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:42.074725 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.065088 30866 raft_consensus.cc:2379] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:43.065572 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1071) took 1093 ms. Trace:
W20260806 20:15:42.188158 30868 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1040) took 1974 ms (client timeout 1577 ms). Trace:
W20260806 20:15:43.067641 30868 rpcz_store.cc:269] 0806 20:15:40.250625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.281574 (+ 30949us) service_pool.cc:227] Handling call
0806 20:15:42.187815 (+1906241us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54002fb7d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:42.188058 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2767721216}
W20260806 20:15:43.068351 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 64217.2us
I20260806 20:15:42.064652 30871 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:43.070001 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1068) took 1116 ms. Trace:
I20260806 20:15:43.070309 30871 rpcz_store.cc:276] 0806 20:15:41.953613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.063712 (+110099us) service_pool.cc:227] Handling call
0806 20:15:43.069934 (+1006222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.071388 30871 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:43.071895 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1069) took 1100 ms. Trace:
I20260806 20:15:43.072160 30871 rpcz_store.cc:276] 0806 20:15:41.971087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.070955 (+1099868us) service_pool.cc:227] Handling call
0806 20:15:43.071228 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.071691 (+ 463us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.071850 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.249334 30703 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:42.251163 30676 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:43.073597 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1070) took 1320 ms. Trace:
W20260806 20:15:43.074088 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 822950us
I20260806 20:15:43.074899 30676 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:42.029224 31414 raft_consensus.cc:1226] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:43.082235 31414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 417) took 1131 ms. Trace:
I20260806 20:15:43.082640 31414 rpcz_store.cc:276] 0806 20:15:41.951124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.953569 (+ 2445us) service_pool.cc:227] Handling call
0806 20:15:42.017496 (+ 63927us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.081939 (+1064443us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.082164 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.075275 32103 raft_consensus.cc:3032] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:43.086558 32103 raft_consensus.cc:729] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:42.100853 32078 consensus_queue.cc:260] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.081411 32105 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:42.250257 32142 tablet_bootstrap.cc:492] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
I20260806 20:15:42.074793 32238 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.099431 32238 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:43.094592 30832 consensus_peers.cc:487] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 -> Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519): Couldn't send request to peer 58f2f6256ee14cb38a1dbcbf5bb590a0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:15:42.084874 32242 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4.
I20260806 20:15:42.068009 32247 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:42.077070 32282 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:42.064873 32283 consensus_queue.cc:1046] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.074s
I20260806 20:15:42.065474 32288 raft_consensus.cc:729] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:43.127588 32288 consensus_queue.cc:260] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.128665 32288 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:42.004534 32311 raft_consensus.cc:729] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:43.140713 32311 consensus_queue.cc:260] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.141840 32311 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:42.066288 32347 consensus_queue.cc:237] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.274492 30708 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:43.147058 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1357) took 1910 ms. Trace:
I20260806 20:15:42.930228 31388 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:43.150492 32271 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:43.150594 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 220454us
I20260806 20:15:42.046929 32396 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:42.270491 32325 tablet_bootstrap.cc:492] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap starting.
W20260806 20:15:42.054153 30791 rpcz_store.cc:269] 0806 20:15:39.997668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.572773 (+575105us) service_pool.cc:227] Handling call
0806 20:15:42.049365 (+1476592us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540014a5d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:42.049527 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2545495296}
I20260806 20:15:43.155810 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:43.154774 32430 consensus_queue.cc:1046] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260806 20:15:43.156407 30791 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:42.313596 30705 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
W20260806 20:15:43.157721 30705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1064) took 1857 ms (client timeout 2173 ms). Trace:
I20260806 20:15:42.065408 30711 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:42.317770 32222 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:43.158681 32222 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.159154 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:43.159809 31417 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:43.160794 32222 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:42.322921 32232 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:42.616397 32339 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:42.652855 32330 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.158061 32430 raft_consensus.cc:3032] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:42.685873 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1451) took 2202 ms. Trace:
I20260806 20:15:42.754324 32149 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:42.822715 32417 log.cc:924] Time spent T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.559s user 0.008s sys 0.000s
I20260806 20:15:43.162042 30786 rpcz_store.cc:276] 0806 20:15:40.483948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:40.968349 (+484401us) service_pool.cc:227] Handling call
0806 20:15:42.685621 (+1717272us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b5400346054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:42.685789 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1603000704}
I20260806 20:15:42.841236 32285 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.163364 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:15:43.164067 30711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1370) took 1244 ms (client timeout 1621 ms). Trace:
I20260806 20:15:42.885677 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1072) took 1130 ms. Trace:
I20260806 20:15:42.891207 32047 consensus_queue.cc:260] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.072669 30789 raft_consensus.cc:2454] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:43.161767 32330 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.165369 30709 rpcz_store.cc:276] 0806 20:15:41.755555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.824801 (+ 69246us) service_pool.cc:227] Handling call
0806 20:15:42.885389 (+1060588us) spinlock_profiling.cc:243] Waited 53.9 ms on lock 0x7b54001b4254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:42.885608 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":124040576}
I20260806 20:15:43.167016 30709 raft_consensus.cc:1261] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260806 20:15:43.167331 30709 pending_rounds.cc:76] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 1
I20260806 20:15:43.167606 30709 pending_rounds.cc:97] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:15:43.167829 30709 raft_consensus.cc:2866] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:42.908253 32345 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:42.919235 30783 consensus_meta.cc:287] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.064s user 0.012s sys 0.006s
I20260806 20:15:42.940565 30713 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.168826 32422 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:43.164402 30711 rpcz_store.cc:269] 0806 20:15:41.920120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.014716 (+ 94596us) service_pool.cc:227] Handling call
0806 20:15:43.164007 (+1149291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.165063 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53732 (request call id 75) took 1154 ms. Trace:
I20260806 20:15:43.169821 30789 rpcz_store.cc:276] 0806 20:15:42.010360 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.033759 (+ 23399us) service_pool.cc:227] Handling call
0806 20:15:43.165001 (+1131242us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:43.158100 30705 rpcz_store.cc:269] 0806 20:15:41.300500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.708084 (+407584us) service_pool.cc:227] Handling call
0806 20:15:41.889225 (+181141us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:43.157649 (+1268424us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2662144}
I20260806 20:15:42.958896 32013 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.170686 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.172353 30705 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:42.960280 30784 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:43.168699 30783 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:43.173053 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1027) took 3717 ms. Trace:
I20260806 20:15:42.960165 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1326) took 4643 ms. Trace:
W20260806 20:15:42.963915 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 922103us
I20260806 20:15:43.173007 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1465) took 1319 ms. Trace:
I20260806 20:15:43.173707 32047 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.173831 30784 rpcz_store.cc:276] 0806 20:15:41.854068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.983038 (+128970us) service_pool.cc:227] Handling call
0806 20:15:43.172942 (+1189904us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.173950 30756 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:43.174916 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.175153 32141 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:42.986711 32322 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:43.174139 30707 rpcz_store.cc:276] 0806 20:15:38.317147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.944433 (+4627286us) service_pool.cc:227] Handling call
0806 20:15:42.944647 (+ 214us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:42.959823 (+ 15176us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:42.959857 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:42.959900 (+ 43us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:42.959928 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:42.959989 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:42.960108 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5514,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":14862,"threads_started":1}
I20260806 20:15:43.176263 32322 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.177117 30707 raft_consensus.cc:1226] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:43.178158 32322 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:42.994112 30787 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:43.004356 31957 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:43.017151 32244 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:43.178618 30713 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:43.040055 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1060) took 1097 ms. Trace:
W20260806 20:15:43.042032 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 1.03776e+06us
I20260806 20:15:43.047608 30869 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
W20260806 20:15:43.173062 30783 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53732 (request call id 71) took 1560 ms (client timeout 1793 ms). Trace:
I20260806 20:15:43.053174 30788 raft_consensus.cc:1261] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:43.179766 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1064) took 1227 ms. Trace:
I20260806 20:15:43.179951 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1452) took 2690 ms. Trace:
I20260806 20:15:43.180126 30869 rpcz_store.cc:276] 0806 20:15:41.952521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.037499 (+1084978us) service_pool.cc:227] Handling call
0806 20:15:43.047408 (+ 9909us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.179521 (+132113us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.179716 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.179160 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1372) took 1258 ms. Trace:
I20260806 20:15:43.181248 30713 rpcz_store.cc:276] 0806 20:15:41.920651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.017602 (+ 96951us) service_pool.cc:227] Handling call
0806 20:15:43.179092 (+1161490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.175277 30756 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:43.182472 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1334) took 4325 ms. Trace:
I20260806 20:15:43.168457 30791 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:43.180483 30788 rpcz_store.cc:276] 0806 20:15:40.489441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.038073 (+1548632us) service_pool.cc:227] Handling call
0806 20:15:42.038356 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.179770 (+1141414us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.179912 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.183866 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1470) took 1097 ms. Trace:
I20260806 20:15:43.184242 30791 rpcz_store.cc:276] 0806 20:15:42.086648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.155524 (+1068876us) service_pool.cc:227] Handling call
0806 20:15:43.183805 (+ 28281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.183259 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1029) took 3726 ms. Trace:
I20260806 20:15:43.172389 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53732 (request call id 76) took 1082 ms. Trace:
I20260806 20:15:43.065665 32369 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:43.066313 30866 rpcz_store.cc:276] 0806 20:15:41.971779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.019978 (+ 48199us) service_pool.cc:227] Handling call
0806 20:15:43.065510 (+1045532us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.066848 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1066) took 1113 ms. Trace:
I20260806 20:15:43.057185 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.069363 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1067) took 1116 ms. Trace:
I20260806 20:15:43.077039 32370 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:43.077503 32337 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:43.193915 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.073966 30703 rpcz_store.cc:276] 0806 20:15:41.752755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.886015 (+133260us) service_pool.cc:227] Handling call
0806 20:15:43.073534 (+1187519us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.080940 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1239) took 1090 ms. Trace:
I20260806 20:15:43.088348 32103 consensus_queue.cc:260] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.092823 32105 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.097597 32078 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:42.018589 32234 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:43.223209 32105 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.109962 30712 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:43.109362 32238 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.113668 32423 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.114840 32247 consensus_queue.cc:1046] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 20:15:43.230087 30712 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41700 (request call id 84) took 5222 ms (client timeout 2141 ms). Trace:
I20260806 20:15:43.229724 32423 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.121233 32223 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.230877 32223 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.231942 32423 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.232944 32223 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.122202 32426 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:43.233863 32426 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.237109 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:43.125422 30863 raft_consensus.cc:2396] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:43.126557 32283 raft_consensus.cc:3032] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:43.135886 32242 consensus_queue.cc:1046] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 20:15:43.137257 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 61224.8us
I20260806 20:15:43.138290 32301 consensus_queue.cc:1035] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:43.141764 32288 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.147423 30708 rpcz_store.cc:276] 0806 20:15:41.236984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.857511 (+620527us) service_pool.cc:227] Handling call
0806 20:15:43.146975 (+1289464us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:42.263301 32388 consensus_queue.cc:237] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:43.147965 31389 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.594s (SENT) [suppressed 18 similar messages]
I20260806 20:15:43.150971 32271 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:43.262878 32271 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.161592 32339 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.163228 32285 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:42.877537 32420 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.168483 32345 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.172117 32013 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:42.965371 32291 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:42.966828 32421 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:42.977329 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:43.175604 32141 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:43.185223 30789 rpcz_store.cc:276] 0806 20:15:42.089577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.170459 (+1080882us) service_pool.cc:227] Handling call
0806 20:15:43.172336 (+ 1877us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.184950 30707 rpcz_store.cc:276] 0806 20:15:39.456555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.176704 (+3720149us) service_pool.cc:227] Handling call
0806 20:15:43.176963 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.183087 (+ 6124us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.183217 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.273972 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:43.274516 30707 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:43.179220 30865 rpcz_store.cc:276] 0806 20:15:41.942930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.024617 (+1081687us) service_pool.cc:227] Handling call
0806 20:15:43.024954 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.039768 (+ 14814us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.039988 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.185963 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1469) took 1099 ms. Trace:
I20260806 20:15:43.187755 30835 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:43.275913 30787 rpcz_store.cc:276] 0806 20:15:42.086354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.992065 (+905711us) service_pool.cc:227] Handling call
0806 20:15:43.185900 (+193835us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:43.276335 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 88665.6us
I20260806 20:15:43.276938 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.182857 30706 rpcz_store.cc:276] 0806 20:15:38.857386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.015974 (+4158588us) service_pool.cc:227] Handling call
0806 20:15:43.016320 (+ 346us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:43.042589 (+ 26269us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.042621 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:43.043103 (+ 482us) log.cc:842] Serialized 26 byte log entry
0806 20:15:43.116925 (+ 73822us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:43.116967 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.117043 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:43.182278 (+ 65235us) raft_consensus.cc:1671] finished
0806 20:15:43.182295 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.182422 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24878,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":4461,"thread_start_us":59318,"threads_started":1,"wal-append.queue_time_us":66186}
I20260806 20:15:43.181854 30757 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:43.278952 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 97174.5us
I20260806 20:15:43.189483 30673 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:15:43.280977 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 91603.3us
I20260806 20:15:43.189121 30756 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:43.164340 30786 raft_consensus.cc:2396] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 20:15:43.281807 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 92782.1us
I20260806 20:15:43.282099 32234 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:43.190580 30869 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:43.283099 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1074) took 1310 ms. Trace:
I20260806 20:15:43.283460 30869 rpcz_store.cc:276] 0806 20:15:41.972646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.190077 (+1217431us) service_pool.cc:227] Handling call
0806 20:15:43.190412 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.282880 (+ 92468us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.283051 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.284356 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.190862 30868 rpcz_store.cc:276] 0806 20:15:41.953328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.068504 (+1115176us) service_pool.cc:227] Handling call
0806 20:15:43.068877 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.069045 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.069070 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.069107 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:43.069134 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.069185 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.069306 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.285619 32428 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.191181 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:43.167255 32330 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.194586 30866 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:43.287089 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1459) took 1990 ms. Trace:
I20260806 20:15:43.194619 30867 rpcz_store.cc:276] 0806 20:15:41.990949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.056560 (+ 65611us) service_pool.cc:227] Handling call
0806 20:15:43.080752 (+1024192us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b540014c3d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:43.080887 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26324736}
I20260806 20:15:43.161973 32430 raft_consensus.cc:729] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:43.194052 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:43.168686 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1343) took 3738 ms. Trace:
I20260806 20:15:43.208664 32232 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:43.285526 30815 debug-util.cc:398] Leaking SignalData structure 0x7b08001c9e60 after lost signal to thread 29422
I20260806 20:15:43.272382 32420 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.300899 32232 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:43.301729 30815 debug-util.cc:398] Leaking SignalData structure 0x7b08001a8aa0 after lost signal to thread 30639
W20260806 20:15:43.302124 30815 debug-util.cc:398] Leaking SignalData structure 0x7b08001a8b60 after lost signal to thread 30655
I20260806 20:15:43.302444 32285 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.302038 32430 consensus_queue.cc:260] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.286187 32141 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.303246 32232 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.208412 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:43.303588 30785 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1471) took 1216 ms (client timeout 1587 ms). Trace:
I20260806 20:15:43.191893 32268 raft_consensus.cc:2741] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:15:43.304402 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.305649 30703 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:43.194319 31957 raft_consensus.cc:686] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:43.305991 30865 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:43.306707 31957 consensus_queue.cc:237] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.308658 30867 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:43.214287 32103 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.162228 32149 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.293994 30709 rpcz_store.cc:276] 0806 20:15:39.430216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.166335 (+3736119us) service_pool.cc:227] Handling call
0806 20:15:43.166682 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.168432 (+ 1750us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.168637 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.188338 30864 rpcz_store.cc:276] 0806 20:15:41.953065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.051085 (+1098020us) service_pool.cc:227] Handling call
0806 20:15:43.066332 (+ 15247us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.066511 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.066538 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.066585 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:43.066614 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.066674 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.066792 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.288231 32223 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:43.214569 32274 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.219591 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1073) took 1247 ms. Trace:
I20260806 20:15:43.120709 30601 catalog_manager.cc:5495] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 2 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:15:43.234761 30712 rpcz_store.cc:269] 0806 20:15:38.007590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.478760 (+471170us) service_pool.cc:227] Handling call
0806 20:15:43.229876 (+4751116us) spinlock_profiling.cc:243] Waited 4.28 s on lock 0x7b5400175054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:43.229993 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9853242112}
I20260806 20:15:43.276001 32337 raft_consensus.cc:686] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:43.303193 32430 raft_consensus.cc:3037] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:43.246234 30676 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:43.333426 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 87826.8us
I20260806 20:15:43.257910 32242 consensus_queue.cc:1225] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer d4181d4d907f477f9feddc44ba3b9207 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:15:43.258879 32426 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.250911 30870 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:43.244019 32283 raft_consensus.cc:729] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:15:43.337834 32433 log.cc:924] Time spent T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260806 20:15:43.291836 32428 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.282497 32234 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:43.286527 30713 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.277158 30835 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:43.356024 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 76017.8us
W20260806 20:15:43.356576 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 79506.2us
I20260806 20:15:43.287878 30710 raft_consensus.cc:2396] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:43.283020 30756 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:43.288540 30789 rpcz_store.cc:276] 0806 20:15:41.297090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.286199 (+1989109us) service_pool.cc:227] Handling call
0806 20:15:43.286506 (+ 307us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.286783 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.286810 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.286850 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:43.286875 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.286929 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.287030 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.289908 30791 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:15:43.357991 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:43.290043 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 20:15:43.304028 30785 rpcz_store.cc:269] 0806 20:15:42.086919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.976876 (+889957us) service_pool.cc:227] Handling call
0806 20:15:43.303520 (+326644us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:43.266508 30863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1072) took 1294 ms (client timeout 1577 ms). Trace:
I20260806 20:15:43.293486 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1236) took 1303 ms. Trace:
I20260806 20:15:43.298434 30707 raft_consensus.cc:2396] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 4 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260806 20:15:43.299132 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 151184us
I20260806 20:15:43.367280 30835 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:43.368088 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1063) took 1416 ms. Trace:
W20260806 20:15:43.303401 30815 debug-util.cc:398] Leaking SignalData structure 0x7b080031b700 after lost signal to thread 30739
I20260806 20:15:43.283370 32238 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:43.179618 30783 rpcz_store.cc:269] 0806 20:15:41.612664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.686341 (+ 73677us) service_pool.cc:227] Handling call
0806 20:15:43.172984 (+1486643us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.311450 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:43.393328 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.395005 30785 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:43.395550 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1210) took 1865 ms. Trace:
I20260806 20:15:43.396093 32337 consensus_queue.cc:237] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.375562 32322 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260806 20:15:43.358552 32234 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.307366 30865 raft_consensus.cc:1226] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:43.395825 30785 rpcz_store.cc:276] 0806 20:15:41.530493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.394530 (+1864037us) service_pool.cc:227] Handling call
0806 20:15:43.394834 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.395355 (+ 521us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.395505 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.186235 30705 rpcz_store.cc:276] 0806 20:15:39.456043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.171857 (+3715814us) service_pool.cc:227] Handling call
0806 20:15:43.172155 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.172810 (+ 655us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.172998 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.245280 32288 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.317024 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:43.319118 30866 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:43.320827 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1457) took 2465 ms. Trace:
I20260806 20:15:43.414093 30787 rpcz_store.cc:276] 0806 20:15:40.855833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.288780 (+2432947us) service_pool.cc:227] Handling call
0806 20:15:43.320118 (+ 31338us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.320476 (+ 358us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.320503 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.320544 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:43.320573 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.320625 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.320761 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":121,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":12136}
I20260806 20:15:43.242707 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:43.373628 32434 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:43.374966 32369 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:43.323844 32435 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:43.376077 32238 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:43.327361 32274 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.328004 30712 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:43.324393 30861 rpcz_store.cc:276] 0806 20:15:41.972362 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.021396 (+1049034us) service_pool.cc:227] Handling call
0806 20:15:43.218715 (+197319us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.301208 31415 raft_consensus.cc:1226] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:43.324920 30601 catalog_manager.cc:5495] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:43.331560 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1070) took 1360 ms. Trace:
I20260806 20:15:43.311529 32347 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:43.338394 32283 consensus_queue.cc:260] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.337495 30864 raft_consensus.cc:1226] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:43.230353 32247 raft_consensus.cc:3032] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:43.273062 32291 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.287652 32421 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.318190 32223 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:43.287393 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1075) took 1314 ms. Trace:
I20260806 20:15:43.368474 30870 rpcz_store.cc:276] 0806 20:15:41.952197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.054408 (+102211us) service_pool.cc:227] Handling call
0806 20:15:43.367929 (+1313521us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54000852d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:43.368033 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2724133248}
I20260806 20:15:43.369846 31389 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:43.367506 30868 rpcz_store.cc:276] 0806 20:15:41.990026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.285786 (+1295760us) service_pool.cc:227] Handling call
0806 20:15:43.292747 (+ 6961us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:43.292980 (+ 233us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.293166 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.293189 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.293228 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:43.293255 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.293308 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.293409 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3666688}
I20260806 20:15:43.372368 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1376) took 1450 ms. Trace:
I20260806 20:15:43.428045 30708 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:43.353961 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:43.455700 32428 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.429663 32435 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.332890 32345 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:43.367228 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 84275us
I20260806 20:15:43.458591 30756 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:43.459592 30756 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:43.460573 30703 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:43.374450 30789 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:43.461776 32232 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:43.370383 30867 raft_consensus.cc:3032] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:43.462368 30867 raft_consensus.cc:729] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:43.374011 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1456) took 2528 ms. Trace:
I20260806 20:15:43.462574 32330 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:43.402165 30788 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:43.462307 32232 raft_consensus.cc:686] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:43.384052 30757 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:43.463989 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 80064.8us
I20260806 20:15:43.463802 32232 consensus_queue.cc:237] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.465238 32330 raft_consensus.cc:686] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:43.465952 32388 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:43.466202 32330 consensus_queue.cc:237] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.462850 30791 rpcz_store.cc:276] 0806 20:15:40.845653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.191348 (+2345695us) service_pool.cc:227] Handling call
0806 20:15:43.191605 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.358516 (+166911us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.373796 (+ 15280us) spinlock_profiling.cc:243] Waited 97.9 ms on lock 0x7b54001d1fd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:43.373953 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225236352}
W20260806 20:15:43.323642 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 115785us
I20260806 20:15:43.468029 32283 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:43.468262 30791 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:15:43.366325 32047 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.438166 32347 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:43.240633 30792 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:43.401979 32282 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.485751 32282 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.487843 32282 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:43.388887 32222 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.488935 32222 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.490339 32347 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:43.490960 32222 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:43.413210 30706 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:43.492482 30706 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:43.423746 30707 raft_consensus.cc:1226] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:43.493019 31388 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:43.493613 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1032) took 3999 ms. Trace:
I20260806 20:15:43.425029 30785 raft_consensus.cc:1226] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:43.493894 32322 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:43.494375 32322 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:43.494603 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1211) took 1963 ms. Trace:
I20260806 20:15:43.494732 32322 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:43.375092 32301 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:43.494947 30785 rpcz_store.cc:276] 0806 20:15:41.530766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.401716 (+1870950us) service_pool.cc:227] Handling call
0806 20:15:43.424868 (+ 23152us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.494365 (+ 69497us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.494551 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.495296 32301 raft_consensus.cc:686] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:43.452363 32223 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:43.496248 32301 consensus_queue.cc:237] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.327018 32149 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.441843 32247 raft_consensus.cc:729] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:43.375692 32268 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:43.430039 30711 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 4. Ops: []
I20260806 20:15:43.498086 32247 consensus_queue.cc:260] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.498869 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1026) took 4043 ms. Trace:
I20260806 20:15:43.499289 32247 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:43.499204 30711 rpcz_store.cc:276] 0806 20:15:39.455732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.169941 (+3714209us) service_pool.cc:227] Handling call
0806 20:15:43.170285 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.498083 (+327798us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.498644 (+ 561us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54000c6f54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:43.498815 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":596938112}
I20260806 20:15:43.389294 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:43.436017 30787 raft_consensus.cc:1261] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:43.500324 30791 raft_consensus.cc:1261] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260806 20:15:43.500700 30791 pending_rounds.cc:76] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Aborting all ops after (but not including) 1
I20260806 20:15:43.500887 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1468) took 1549 ms. Trace:
I20260806 20:15:43.501269 30787 rpcz_store.cc:276] 0806 20:15:41.951058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.415291 (+1464233us) service_pool.cc:227] Handling call
0806 20:15:43.435705 (+ 20414us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.500655 (+ 64950us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.500834 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.373991 32242 consensus_queue.cc:1035] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260806 20:15:43.437836 30871 rpcz_store.cc:276] 0806 20:15:41.971458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.072783 (+1101325us) service_pool.cc:227] Handling call
0806 20:15:43.073039 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.287837 (+214798us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.287866 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.287904 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:43.287932 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.331299 (+ 43367us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540019da54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:43.331353 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.331491 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":473459968}
I20260806 20:15:43.290731 32370 raft_consensus.cc:686] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:15:43.434005 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 94632.5us
I20260806 20:15:43.437299 30601 catalog_manager.cc:5495] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 2 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:43.400115 32285 consensus_queue.cc:1046] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260806 20:15:43.427969 32369 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:43.411314 32141 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:43.370751 30863 rpcz_store.cc:269] 0806 20:15:41.972095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.031480 (+ 59385us) service_pool.cc:227] Handling call
0806 20:15:43.261331 (+1229851us) spinlock_profiling.cc:243] Waited 87.3 ms on lock 0x7b54002fb7d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:43.266132 (+ 4801us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":200776448}
I20260806 20:15:43.445739 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1033) took 3950 ms. Trace:
W20260806 20:15:43.437166 30815 debug-util.cc:398] Leaking SignalData structure 0x7b0800353c60 after lost signal to thread 30818
I20260806 20:15:43.434749 32238 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.427944 32337 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:43.449980 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 80215.5us
I20260806 20:15:43.566936 31389 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:43.365870 32426 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:43.449837 30710 rpcz_store.cc:276] 0806 20:15:41.921648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:42.064195 (+142547us) service_pool.cc:227] Handling call
0806 20:15:43.372178 (+1307983us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54001b4754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:43.372314 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":657475456}
I20260806 20:15:43.582093 32426 raft_consensus.cc:686] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:43.582737 32234 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.407724 30713 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:43.584535 30713 raft_consensus.cc:1226] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:43.585059 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1048) took 3550 ms. Trace:
I20260806 20:15:43.585327 30713 rpcz_store.cc:276] 0806 20:15:40.034307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.584123 (+3549816us) service_pool.cc:227] Handling call
0806 20:15:43.584387 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.584870 (+ 483us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.585013 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.467296 32421 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.462006 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 68) took 1936 ms. Trace:
I20260806 20:15:43.465337 30757 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:43.374507 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:43.477983 32047 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:43.469856 30867 consensus_queue.cc:260] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.479185 32288 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.480705 32274 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.452760 32339 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.484987 32291 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.512144 32420 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.469058 32423 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:43.493048 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1040) took 3918 ms. Trace:
I20260806 20:15:43.500907 30791 pending_rounds.cc:97] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:15:43.493960 30707 rpcz_store.cc:276] 0806 20:15:39.494502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.371350 (+3876848us) service_pool.cc:227] Handling call
0806 20:15:43.423421 (+ 52071us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:43.423622 (+ 201us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.493370 (+ 69748us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.493564 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7168768}
I20260806 20:15:43.505017 30676 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:43.506070 30601 catalog_manager.cc:5495] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:15:43.509135 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 86124.4us
I20260806 20:15:43.469724 32443 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:43.563140 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 119096us
I20260806 20:15:43.563575 30787 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:43.593825 32345 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:43.605120 32443 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.504868 32268 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.509006 32369 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:43.587553 32337 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.606395 32268 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.607228 32443 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:43.595134 30791 raft_consensus.cc:2866] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:43.608007 32337 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.608351 32268 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:43.596235 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 91323.6us
I20260806 20:15:43.563621 32422 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.513517 30705 rpcz_store.cc:276] 0806 20:15:39.494886 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.427295 (+3932409us) service_pool.cc:227] Handling call
0806 20:15:43.445203 (+ 17908us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.445377 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.445403 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.445488 (+ 85us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:43.445518 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.445572 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.445680 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.571544 32142 tablet_bootstrap.cc:492] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:15:43.577281 30708 raft_consensus.cc:1226] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:43.577636 30703 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:43.578058 30786 raft_consensus.cc:1226] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:15:43.579252 32434 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:43.579814 32370 consensus_queue.cc:237] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.583063 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1047) took 3549 ms. Trace:
I20260806 20:15:43.449308 30869 rpcz_store.cc:276] 0806 20:15:41.972951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.285285 (+1312334us) service_pool.cc:227] Handling call
0806 20:15:43.286954 (+ 1669us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.287116 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.287140 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.287185 (+ 45us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:43.287212 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.287257 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.287346 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.586714 30789 rpcz_store.cc:276] 0806 20:15:41.525861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.372802 (+1846941us) service_pool.cc:227] Handling call
0806 20:15:43.373130 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.461720 (+ 88590us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.461948 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.587715 30711 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:43.458504 32271 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.586622 32325 tablet_bootstrap.cc:492] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260806 20:15:43.589291 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:43.589663 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 124406us
W20260806 20:15:43.590444 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 121962us
I20260806 20:15:43.595041 30706 rpcz_store.cc:276] 0806 20:15:39.574444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.491998 (+3917554us) service_pool.cc:227] Handling call
0806 20:15:43.492323 (+ 325us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.492844 (+ 521us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.493000 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.596122 30707 raft_consensus.cc:1226] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:43.599444 32445 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:43.596187 30601 catalog_manager.cc:5495] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:43.601750 32047 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.602349 30785 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:15:43.586719 32234 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:43.605980 30815 debug-util.cc:398] Leaking SignalData structure 0x7b08003fba80 after lost signal to thread 30897
I20260806 20:15:43.594156 30709 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:43.605922 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.608659 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.575376 32347 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:43.650104 30792 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1178) took 5272 ms (client timeout 1708 ms). Trace:
I20260806 20:15:43.650630 30790 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 20:15:43.795717 30792 rpcz_store.cc:269] 0806 20:15:38.378589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:38.927313 (+548724us) service_pool.cc:227] Handling call
0806 20:15:43.649883 (+4722570us) spinlock_profiling.cc:243] Waited 4.21 s on lock 0x7b5400000d54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:43.650006 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9693444480}
I20260806 20:15:43.797065 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.638321 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1455) took 2810 ms. Trace:
I20260806 20:15:43.753110 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1351) took 3570 ms. Trace:
I20260806 20:15:43.797804 30792 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:43.798194 30707 rpcz_store.cc:276] 0806 20:15:40.183182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.595684 (+3412502us) service_pool.cc:227] Handling call
0806 20:15:43.595964 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.750232 (+154268us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.753058 (+ 2826us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.757380 30713 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:43.800560 30788 rpcz_store.cc:276] 0806 20:15:40.828078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.190204 (+2362126us) service_pool.cc:227] Handling call
0806 20:15:43.190478 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.463184 (+272706us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.638062 (+174878us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54001d24d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:43.638256 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":485356288}
I20260806 20:15:43.619894 30866 raft_consensus.cc:1226] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:15:43.797737 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1044) took 4187 ms. Trace:
I20260806 20:15:43.620313 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.621039 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.749200 30601 catalog_manager.cc:5495] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 2 to 3, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 3 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:43.802742 30711 rpcz_store.cc:276] 0806 20:15:39.610124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.500126 (+3890002us) service_pool.cc:227] Handling call
0806 20:15:43.587549 (+ 87423us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.749976 (+162427us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.797530 (+ 47554us) spinlock_profiling.cc:243] Waited 87.1 ms on lock 0x7b54002120d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:43.797663 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":200430848}
I20260806 20:15:43.803933 30601 catalog_manager.cc:5495] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 4, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 4 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:43.751355 30757 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:43.806130 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 54861.1us
I20260806 20:15:43.808650 30601 catalog_manager.cc:5495] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 2 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:43.810644 30601 catalog_manager.cc:5495] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:43.812320 30601 catalog_manager.cc:5495] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 2, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 2 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:43.617127 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.749540 30710 rpcz_store.cc:276] 0806 20:15:40.034043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.582229 (+3548186us) service_pool.cc:227] Handling call
0806 20:15:43.582579 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.582752 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.582775 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.582832 (+ 57us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:43.582859 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.582907 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.583010 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:43.617507 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.817643 30868 raft_consensus.cc:2379] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:43.815359 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.819552 30601 catalog_manager.cc:5495] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 5, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 5 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:43.592500 32396 raft_consensus.cc:2781] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:43.823874 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.818612 30865 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:43.830003 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.830752 30865 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:43.756605 30787 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:43.833521 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1199) took 3780 ms. Trace:
I20260806 20:15:43.833848 30790 rpcz_store.cc:276] 0806 20:15:40.052958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:41.705318 (+1652360us) service_pool.cc:227] Handling call
0806 20:15:43.650293 (+1944975us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.795341 (+145048us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.833321 (+ 37980us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b5400000d50. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:43.833467 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4462998784}
I20260806 20:15:43.834414 30835 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:43.835166 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.833169 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1052) took 3797 ms. Trace:
I20260806 20:15:43.835793 30713 rpcz_store.cc:276] 0806 20:15:40.035377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.586013 (+3550636us) service_pool.cc:227] Handling call
0806 20:15:43.586310 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.800708 (+214398us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.832932 (+ 32224us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54000a87d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:43.833109 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":392975104}
I20260806 20:15:43.834321 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.837145 30788 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:43.838059 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.825045 30871 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:43.844921 30788 raft_consensus.cc:2379] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:43.700495 32388 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:43.845782 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 256540us
I20260806 20:15:43.846659 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:43.849553 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 54404.6us
I20260806 20:15:43.852288 30834 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:43.656204 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 72) took 1827 ms. Trace:
I20260806 20:15:43.616632 32420 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.636255 32013 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.614566 32141 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.717933 32283 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.658203 32103 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.714038 32447 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:43.749130 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.857021 32443 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:43.619158 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.862946 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.928491 30868 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:43.755980 32345 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.864488 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.620611 32337 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:43.611176 32435 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.853551 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.865321 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.749540 32445 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:43.592118 31957 consensus_queue.cc:1046] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.255s
I20260806 20:15:43.833422 32274 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:43.833177 32428 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.714135 32339 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.144s
I20260806 20:15:43.844609 32242 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.733811 32244 consensus_queue.cc:1035] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.132s
I20260806 20:15:43.634287 32430 consensus_queue.cc:1046] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.232s
I20260806 20:15:43.733930 32222 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.132s
I20260806 20:15:43.693948 32369 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.656234 32238 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.652853 32322 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.700376 32223 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.870473 31416 raft_consensus.cc:1226] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:43.832367 32047 consensus_queue.cc:1046] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.085s
I20260806 20:15:43.868230 32282 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:43.863875 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:43.583089 32426 consensus_queue.cc:237] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.982802 32271 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:43.983105 32452 log.cc:924] Time spent T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260806 20:15:43.967690 32337 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.847872 32234 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260806 20:15:43.984225 32238 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.984948 32337 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.985517 32234 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.985849 32271 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:15:43.986094 32234 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.986986 32223 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:43.922576 32141 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:43.924296 32443 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:43.958446 32325 tablet_bootstrap.cc:492] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:43.866966 32388 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.988659 32325 ts_tablet_manager.cc:1397] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 1.718s user 0.017s sys 0.036s
I20260806 20:15:43.962826 32142 tablet_bootstrap.cc:492] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Bootstrap complete.
I20260806 20:15:43.869192 32420 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.919124 32283 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.920212 32103 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.969905 32445 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.960251 32454 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:43.987993 32443 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:43.801883 32422 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.931094 32447 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:43.873278 32448 consensus_queue.cc:1046] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.165s
W20260806 20:15:44.013901 32247 consensus_meta.cc:287] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.509s user 0.006s sys 0.002s
I20260806 20:15:43.962255 32345 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.982453 32423 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:43.985313 32282 raft_consensus.cc:686] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:44.008476 32388 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.014564 32396 raft_consensus.cc:686] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:44.021198 32369 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:44.021380 32322 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.021678 32448 consensus_queue.cc:1225] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer d4181d4d907f477f9feddc44ba3b9207 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 20:15:44.022051 32347 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.022593 32420 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.024680 32396 consensus_queue.cc:237] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.023433 32345 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:43.976495 32428 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.028667 32428 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:44.030421 32444 raft_consensus.cc:3032] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:44.018198 32454 raft_consensus.cc:686] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:44.032680 32422 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.030874 32444 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:44.032892 32454 consensus_queue.cc:237] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.033478 32447 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:44.034675 32444 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.035619 32444 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:43.821208 30601 catalog_manager.cc:5495] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 3, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 3 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:43.854596 30706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e"
dest_uuid: "42cabca48db7403b8523a3382a046043"
mode: GRACEFUL
new_leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
from {username='slave'} at 127.0.0.1:38516
I20260806 20:15:43.862254 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1034) took 4354 ms. Trace:
I20260806 20:15:44.071185 30706 raft_consensus.cc:595] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Received request to transfer leadership to TS 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:44.073546 30705 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:44.074635 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1352) took 3878 ms. Trace:
I20260806 20:15:44.074389 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.075150 30705 rpcz_store.cc:276] 0806 20:15:40.196364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.802172 (+3605808us) service_pool.cc:227] Handling call
0806 20:15:44.072739 (+270567us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:44.074233 (+ 1494us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.074518 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.075911 30706 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:44.071723 30703 rpcz_store.cc:276] 0806 20:15:39.495139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.476187 (+3981048us) service_pool.cc:227] Handling call
0806 20:15:43.563646 (+ 87459us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.756749 (+193103us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.849074 (+ 92325us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.079437 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.080431 30703 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:44.081911 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.082947 30703 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:44.084272 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.085625 30703 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:44.077793 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:44.077183 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:43.874631 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1057) took 3379 ms. Trace:
I20260806 20:15:43.869944 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1056) took 3377 ms. Trace:
W20260806 20:15:43.876857 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 54118.2us
I20260806 20:15:44.117846 30711 rpcz_store.cc:276] 0806 20:15:40.492485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.833282 (+3340797us) service_pool.cc:227] Handling call
0806 20:15:43.869424 (+ 36142us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.869626 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.869653 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.869692 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:43.869720 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.869767 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.869884 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.119058 30757 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:43.616328 30784 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:44.122424 32422 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:43.854899 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.127092 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.128123 30785 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:44.128229 32282 consensus_queue.cc:237] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:43.917629 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.871965 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.920678 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.132440 30835 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.134101 32222 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:43.868688 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.134677 32222 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:44.135198 32222 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:44.135413 30790 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
W20260806 20:15:43.922888 30815 debug-util.cc:398] Leaking SignalData structure 0x7b08003ac4e0 after lost signal to thread 31449
I20260806 20:15:44.136478 32388 consensus_queue.cc:1046] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 20:15:43.971068 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 116901us
I20260806 20:15:44.138417 32388 raft_consensus.cc:3032] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:44.139060 32388 raft_consensus.cc:729] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:44.139549 31957 consensus_queue.cc:1035] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:43.621516 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:44.141768 30863 raft_consensus.cc:2379] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:44.142102 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:43.620716 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:43.853786 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:44.091589 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1461) took 2777 ms. Trace:
I20260806 20:15:44.144112 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.144764 30786 rpcz_store.cc:276] 0806 20:15:41.314968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.282235 (+1967267us) service_pool.cc:227] Handling call
0806 20:15:43.282585 (+ 350us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:43.756291 (+473706us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.923010 (+166719us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b540026e654. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:44.091233 (+168223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":655776512}
I20260806 20:15:44.145787 30705 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:44.146020 32234 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:44.146224 30785 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:44.146893 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.147450 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.148005 30786 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:44.091782 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:43.987942 32141 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.110404 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:43.863473 30783 rpcz_store.cc:276] 0806 20:15:41.828449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.411877 (+1583428us) service_pool.cc:227] Handling call
0806 20:15:43.437986 (+ 26109us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:43.501079 (+ 63093us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.501113 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:43.501609 (+ 496us) log.cc:842] Serialized 26 byte log entry
0806 20:15:43.598206 (+ 96597us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:15:43.598565 (+ 359us) log.cc:842] Serialized 14 byte log entry
0806 20:15:43.598800 (+ 235us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.655770 (+ 56970us) spinlock_profiling.cc:243] Waited 61 ms on lock 0x7b54003d45d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:43.655826 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:43.655875 (+ 49us) raft_consensus.cc:1671] finished
0806 20:15:43.655971 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.656121 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":616,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":983,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":187,"spinlock_wait_cycles":140407168,"thread_start_us":96194,"threads_started":1,"wal-append.queue_time_us":96559}
I20260806 20:15:44.151649 32141 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.108163 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:43.976631 32242 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.152813 30783 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:43.855618 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1046) took 3821 ms. Trace:
I20260806 20:15:44.113421 32421 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:44.153430 30791 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:44.154065 32421 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:44.114118 30710 rpcz_store.cc:276] 0806 20:15:40.495118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.831769 (+3336651us) service_pool.cc:227] Handling call
0806 20:15:43.874264 (+ 42495us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.874397 (+ 133us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.874412 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:43.874446 (+ 34us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:43.874471 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.874512 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:43.874587 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.091933 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.140702 32388 consensus_queue.cc:260] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.140754 30834 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:44.012029 32268 consensus_queue.cc:1046] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.007s
I20260806 20:15:44.149240 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.157647 30703 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:44.158310 31414 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:44.157406 32421 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.171096 32013 raft_consensus.cc:3032] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:44.171607 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.172585 30706 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:44.110704 32142 ts_tablet_manager.cc:1397] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent bootstrapping tablet: real 1.861s user 0.038s sys 0.022s
I20260806 20:15:44.165390 30785 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:44.164899 30792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1474) took 1022 ms. Trace:
I20260806 20:15:44.179111 30792 rpcz_store.cc:276] 0806 20:15:43.138502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.159161 (+1020659us) service_pool.cc:227] Handling call
0806 20:15:44.159889 (+ 728us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.160425 (+ 536us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:44.160487 (+ 62us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:44.160563 (+ 76us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:44.160600 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.160680 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:44.160786 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.162746 30713 raft_consensus.cc:1226] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:44.167591 30786 raft_consensus.cc:2396] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:44.168672 30834 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:44.163225 32244 consensus_queue.cc:1046] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260806 20:15:44.163388 32242 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:44.164882 30708 rpcz_store.cc:276] 0806 20:15:40.033746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.509857 (+3476111us) service_pool.cc:227] Handling call
0806 20:15:43.577057 (+ 67200us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.755966 (+178909us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.850058 (+ 94092us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.165745 32234 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:44.193095 32234 raft_consensus.cc:686] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:44.194653 30708 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:44.195087 32234 consensus_queue.cc:237] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.171631 32013 raft_consensus.cc:729] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:44.189073 30867 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:44.164546 32283 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:44.198153 30791 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:44.198184 32013 consensus_queue.cc:260] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.202574 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1031) took 4740 ms. Trace:
I20260806 20:15:44.203078 32013 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:15:44.203437 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 352225us
I20260806 20:15:44.203877 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.204833 30789 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:44.206073 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.190533 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:44.208436 32149 raft_consensus.cc:3032] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:44.208946 32149 raft_consensus.cc:729] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:44.209635 30789 raft_consensus.cc:1261] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:44.211351 32103 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:44.184301 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 79) took 1119 ms. Trace:
I20260806 20:15:44.211576 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1478) took 1070 ms. Trace:
I20260806 20:15:44.210033 32244 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:44.213059 32422 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:44.213806 32396 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:15:44.216638 32454 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:44.217042 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:44.203085 30712 rpcz_store.cc:276] 0806 20:15:39.461776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.323218 (+3861442us) service_pool.cc:227] Handling call
0806 20:15:43.323497 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.435052 (+111555us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:43.647626 (+212574us) spinlock_profiling.cc:243] Waited 78.7 ms on lock 0x7b54003c49a0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71515ff730 00007f7151672b42 00007f715166ef26 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:15:44.201947 (+554321us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54003c49a0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f7151604cbd 00007f7151672b97 00007f715166ef26 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:15:44.202350 (+ 403us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1228128896}
I20260806 20:15:44.218786 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:44.218987 32345 consensus_queue.cc:1046] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:15:44.221828 30787 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:44.210559 32149 consensus_queue.cc:260] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.235894 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1475) took 1096 ms. Trace:
I20260806 20:15:44.236280 32149 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:44.236644 31414 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:44.224831 30789 rpcz_store.cc:276] 0806 20:15:43.140970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.208376 (+1067406us) service_pool.cc:227] Handling call
0806 20:15:44.208962 (+ 586us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.211217 (+ 2255us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.211481 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.239162 30789 raft_consensus.cc:1226] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:44.231498 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.239449 30792 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:44.232910 31416 raft_consensus.cc:2379] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 2: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:44.238796 30784 rpcz_store.cc:276] 0806 20:15:43.139114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.174640 (+ 35526us) service_pool.cc:227] Handling call
0806 20:15:44.235511 (+1060871us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b54001d15d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:44.235688 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1013519616}
I20260806 20:15:44.210994 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1364) took 2972 ms. Trace:
I20260806 20:15:44.242187 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 88) took 3285 ms. Trace:
I20260806 20:15:44.242594 30713 rpcz_store.cc:276] 0806 20:15:40.956922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.836663 (+2879741us) service_pool.cc:227] Handling call
0806 20:15:43.863991 (+ 27328us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.191760 (+327769us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.241914 (+ 50154us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400175a54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:44.242121 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":549159168}
I20260806 20:15:44.243861 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:44.238698 32463 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.050s user 0.002s sys 0.000s
I20260806 20:15:44.245955 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.246793 30790 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:44.247570 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:44.249354 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:44.250352 30709 raft_consensus.cc:1261] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:44.250013 32443 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.251005 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1363) took 3012 ms. Trace:
I20260806 20:15:44.251405 30709 rpcz_store.cc:276] 0806 20:15:41.238600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.249349 (+3010749us) service_pool.cc:227] Handling call
0806 20:15:44.249890 (+ 541us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.250726 (+ 836us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.250932 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.252236 32443 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:44.252769 32462 log.cc:924] Time spent T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.188s user 0.001s sys 0.000s
I20260806 20:15:44.252494 32325 raft_consensus.cc:348] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.245177 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.258414 30713 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:44.258738 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:44.253232 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.261540 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.262387 30786 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:44.263430 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.264367 30703 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:44.242321 30708 rpcz_store.cc:276] 0806 20:15:41.238863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.192931 (+2954068us) service_pool.cc:227] Handling call
0806 20:15:44.194104 (+ 1173us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.199926 (+ 5822us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.210883 (+ 10957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.265352 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.266690 30703 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:44.266970 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:44.272320 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:44.275460 30787 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:44.279291 32345 raft_consensus.cc:3032] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:44.279786 32345 raft_consensus.cc:729] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:44.281946 30708 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:44.280740 32345 consensus_queue.cc:260] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.262492 30705 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:44.284674 30703 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:15:44.226070 30783 rpcz_store.cc:276] 0806 20:15:43.062092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.151767 (+1089675us) service_pool.cc:227] Handling call
0806 20:15:44.152231 (+ 464us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.153256 (+ 1025us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.181439 (+ 28183us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:44.300400 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 71234.2us
I20260806 20:15:44.305905 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:44.306941 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.309289 31957 raft_consensus.cc:3032] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:44.310055 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1055) took 4014 ms. Trace:
W20260806 20:15:44.310842 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 138402us
I20260806 20:15:44.313382 31957 raft_consensus.cc:729] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:44.323189 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1367) took 2978 ms. Trace:
I20260806 20:15:44.323040 31957 consensus_queue.cc:260] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.323632 30709 rpcz_store.cc:276] 0806 20:15:41.344876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.310490 (+2965614us) service_pool.cc:227] Handling call
0806 20:15:44.322193 (+ 11703us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.322738 (+ 545us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:44.322772 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:44.322841 (+ 69us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:44.322885 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.322946 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:44.323090 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.324630 31957 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:44.326378 30711 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:44.327421 30867 raft_consensus.cc:1261] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260806 20:15:44.330518 32078 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260806 20:15:44.331493 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1368) took 2981 ms. Trace:
I20260806 20:15:44.331966 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:44.330672 30709 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:44.328395 30785 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:44.330533 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1079) took 1139 ms. Trace:
I20260806 20:15:44.327111 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 94) took 3011 ms. Trace:
I20260806 20:15:44.252162 32142 raft_consensus.cc:348] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:44.336475 30711 rpcz_store.cc:276] 0806 20:15:41.316195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.289223 (+2973028us) service_pool.cc:227] Handling call
0806 20:15:44.299286 (+ 10063us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.326836 (+ 27550us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.327035 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.337435 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1477) took 1197 ms. Trace:
I20260806 20:15:44.346657 30785 rpcz_store.cc:276] 0806 20:15:43.140391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.184273 (+1043882us) service_pool.cc:227] Handling call
0806 20:15:44.273922 (+ 89649us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.337147 (+ 63225us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.337358 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.338341 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:44.343307 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 490424us
I20260806 20:15:44.342157 31423 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:44.342553 32445 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.335942 30867 rpcz_store.cc:276] 0806 20:15:43.191547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.309388 (+117841us) service_pool.cc:227] Handling call
0806 20:15:43.309668 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:43.370286 (+ 60618us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b54002ce0a0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71515ff730 00007f7151671300 00007f7151672371 00007f715166eedf 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc
0806 20:15:43.637561 (+267275us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b58000d25b0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71515eaf63 00007f715164eaba 00007f71516663af 00007f7151667b4e 00007f7151671c89 00007f7151672556 00007f715166eedf 00007f715166e99e 00007f71565b02e9 00007f714fca2684
0806 20:15:44.180035 (+542474us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54002ce0a0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71515fd40d 00007f71515eb050 00007f715164eaba 00007f71516663af 00007f7151667b4e 00007f7151671c89 00007f7151672556 00007f715166eedf 00007f715166e99e 00007f71565b02e9
0806 20:15:44.327797 (+147762us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.330446 (+ 2649us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1140356480}
I20260806 20:15:44.337347 30788 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:44.331914 30706 rpcz_store.cc:276] 0806 20:15:41.349841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.327113 (+2977272us) service_pool.cc:227] Handling call
0806 20:15:44.330722 (+ 3609us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.331113 (+ 391us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:44.331150 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:44.331229 (+ 79us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:44.331261 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.331322 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:44.331436 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":127,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":388}
I20260806 20:15:44.312032 30707 rpcz_store.cc:276] 0806 20:15:40.284225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.799533 (+3515308us) service_pool.cc:227] Handling call
0806 20:15:43.799913 (+ 380us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:43.825596 (+ 25683us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:43.825630 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:43.826121 (+ 491us) log.cc:842] Serialized 26 byte log entry
0806 20:15:44.189643 (+363522us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:44.190869 (+ 1226us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.207932 (+ 17063us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:44.298206 (+ 90274us) raft_consensus.cc:1671] finished
0806 20:15:44.298230 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:44.298403 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25273,"raft.queue_time_us":26132,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":24743,"thread_start_us":306648,"threads_started":1,"wal-append.queue_time_us":408686}
I20260806 20:15:44.329697 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1365) took 3089 ms. Trace:
W20260806 20:15:44.352854 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 55815.8us
I20260806 20:15:44.356372 30711 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:44.346423 32142 raft_consensus.cc:729] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:44.357085 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 100) took 1478 ms. Trace:
W20260806 20:15:44.354735 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 186372us
I20260806 20:15:44.360347 30711 rpcz_store.cc:276] 0806 20:15:42.878484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.350528 (+1472044us) service_pool.cc:227] Handling call
0806 20:15:44.355884 (+ 5356us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.356799 (+ 915us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.357033 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.362238 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:44.364444 30710 raft_consensus.cc:1226] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:44.365823 30707 raft_consensus.cc:1261] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:44.366513 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1378) took 1185 ms. Trace:
I20260806 20:15:44.366837 30707 rpcz_store.cc:276] 0806 20:15:43.181264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.364952 (+1183688us) service_pool.cc:227] Handling call
0806 20:15:44.365396 (+ 444us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.366265 (+ 869us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.366466 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.368592 30707 raft_consensus.cc:1261] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:44.369123 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1382) took 1185 ms. Trace:
I20260806 20:15:44.350716 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1071) took 2596 ms. Trace:
I20260806 20:15:44.366889 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:44.372521 30706 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:15:44.352118 30703 rpcz_store.cc:276] 0806 20:15:41.239129 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.267590 (+3028461us) service_pool.cc:227] Handling call
0806 20:15:44.270517 (+ 2927us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.311550 (+ 41033us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.328965 (+ 17415us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.373888 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:44.371341 30711 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:44.382961 32013 consensus_queue.cc:1046] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:44.401300 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 101) took 1468 ms. Trace:
I20260806 20:15:44.403131 30706 rpcz_store.cc:276] 0806 20:15:42.932505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.371559 (+1439054us) service_pool.cc:227] Handling call
0806 20:15:44.372041 (+ 482us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.400970 (+ 28929us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.401245 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.383953 30756 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.348049 32222 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.388511 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 91) took 3217 ms. Trace:
I20260806 20:15:44.389648 32345 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:44.390558 30866 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:15:44.391110 30865 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:44.402273 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 102) took 1220 ms. Trace:
I20260806 20:15:44.393472 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:44.393851 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:44.394753 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:44.396306 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:44.398428 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:44.400471 30707 rpcz_store.cc:276] 0806 20:15:43.183878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.367816 (+1183938us) service_pool.cc:227] Handling call
0806 20:15:44.368188 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.368912 (+ 724us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.369076 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.422966 30711 rpcz_store.cc:276] 0806 20:15:43.182131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.370649 (+1188518us) service_pool.cc:227] Handling call
0806 20:15:44.371016 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.401983 (+ 30967us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.402214 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.354974 32325 raft_consensus.cc:729] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Initialized, Role: LEARNER
I20260806 20:15:44.422435 30710 rpcz_store.cc:276] 0806 20:15:41.171499 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.258537 (+3087038us) service_pool.cc:227] Handling call
0806 20:15:44.258986 (+ 449us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.364855 (+105869us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.388243 (+ 23388us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54001b3d54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:44.388432 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":234600320}
I20260806 20:15:44.428148 30788 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:44.437712 30834 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:44.425026 32078 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:44.424728 32422 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:44.442847 30703 raft_consensus.cc:1261] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:15:44.443471 30870 raft_consensus.cc:2379] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:44.447209 30706 raft_consensus.cc:1261] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:15:44.401561 30709 rpcz_store.cc:276] 0806 20:15:41.752997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.324751 (+2571754us) service_pool.cc:227] Handling call
0806 20:15:44.325213 (+ 462us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.334804 (+ 9591us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.349574 (+ 14770us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.449481 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1383) took 1265 ms. Trace:
I20260806 20:15:44.451308 30706 rpcz_store.cc:276] 0806 20:15:43.184158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.425369 (+1241211us) service_pool.cc:227] Handling call
0806 20:15:44.446872 (+ 21503us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.447588 (+ 716us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.449405 (+ 1817us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.446076 30835 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.455552 32370 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:44.463148 30710 raft_consensus.cc:1261] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:44.462896 32370 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.450251 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:44.445963 32422 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:44.398500 32142 consensus_queue.cc:260] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:44.446375 30861 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:44.475765 30756 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.476027 32222 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:44.455243 32078 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.465878 32422 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:44.479565 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:44.480365 30709 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:44.481969 32370 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:44.464637 30757 leader_election.cc:334] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.577s (SENT)
W20260806 20:15:44.487310 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 60563.5us
I20260806 20:15:44.491214 32222 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:44.491897 32222 raft_consensus.cc:686] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:44.492465 30756 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.493155 32222 consensus_queue.cc:237] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.493675 32141 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:44.494182 32141 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:44.488904 30711 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:44.494599 32141 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:44.460145 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1384) took 1266 ms. Trace:
I20260806 20:15:44.494905 32467 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260806 20:15:44.493813 30757 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:44.496671 30756 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.494671 30708 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:44.497722 32013 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:44.494310 32274 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:44.498562 32274 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:44.498947 32274 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:44.499164 32078 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:44.501863 30712 raft_consensus.cc:2396] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:44.494932 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 104) took 1054 ms. Trace:
I20260806 20:15:44.504274 30711 rpcz_store.cc:276] 0806 20:15:43.440567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.435583 (+995016us) service_pool.cc:227] Handling call
0806 20:15:44.459950 (+ 24367us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.494690 (+ 34740us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.494879 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.525245 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1375) took 2603 ms. Trace:
I20260806 20:15:44.525799 30705 rpcz_store.cc:276] 0806 20:15:41.921358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.346534 (+2425176us) service_pool.cc:227] Handling call
0806 20:15:44.390902 (+ 44368us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.391302 (+ 400us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:44.434760 (+ 43458us) log.cc:842] Serialized 14 byte log entry
0806 20:15:44.520208 (+ 85448us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:44.522908 (+ 2700us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:44.522999 (+ 91us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.524985 (+ 1986us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:44.525156 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18621,"prepare.queue_time_us":82914,"raft.queue_time_us":59958,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":142,"spinlock_wait_cycles":8701824,"thread_start_us":60426,"threads_started":2}
I20260806 20:15:44.527208 32142 ts_tablet_manager.cc:1428] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.305s user 0.007s sys 0.004s
I20260806 20:15:44.528275 30676 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:44.528306 32149 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260806 20:15:44.503517 32141 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.531896 32322 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:44.493716 32388 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:44.532953 32423 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:44.534603 32423 consensus_queue.cc:1046] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:15:44.535676 32454 raft_consensus.cc:3032] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:44.536048 32454 raft_consensus.cc:729] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:44.485076 32325 consensus_queue.cc:260] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:44.537045 32454 consensus_queue.cc:260] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.537854 32325 ts_tablet_manager.cc:1428] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent starting tablet: real 0.549s user 0.011s sys 0.002s
W20260806 20:15:44.538200 32325 ts_tablet_manager.cc:1460] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Tablet startup took 549ms
W20260806 20:15:44.538527 32325 ts_tablet_manager.cc:1462] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Trace:
0806 20:15:36.421505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:36.421893 (+ 388us) service_pool.cc:227] Handling call
0806 20:15:37.989552 (+1567659us) spinlock_profiling.cc:243] Waited 3.93 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f714e7355fe 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71565fbda2
0806 20:15:38.925709 (+936157us) ts_tablet_manager.cc:928] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Initiating tablet copy from peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937)
0806 20:15:40.163105 (+1237396us) tablet_copy_client.cc:361] Tablet copy session begun
0806 20:15:41.257434 (+1094329us) tablet_metadata.cc:641] Metadata flushed
0806 20:15:41.257800 (+ 366us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 20:15:41.527608 (+269808us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 20:15:41.528000 (+ 392us) tablet_replica.cc:205] Creating consensus instance
0806 20:15:41.567717 (+ 39717us) ts_tablet_manager.cc:983] Replica registered.
0806 20:15:41.602566 (+ 34849us) inbound_call.cc:173] Queueing success response
0806 20:15:42.270067 (+667501us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0806 20:15:43.989049 (+1718982us) ts_tablet_manager.cc:1429] Starting tablet replica
0806 20:15:44.114121 (+125072us) tablet_replica.cc:259] Starting instrumentation
0806 20:15:44.115179 (+ 1058us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":155838,"raft.queue_time_us":1508,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"spinlock_wait_cycles":9035648,"tablet-copy.queue_time_us":2444223,"tablet-download-pool.queue_time_us":2190,"tablet-download-pool.run_cpu_time_us":1386,"tablet-download-pool.run_wall_time_us":244282,"thread_start_us":2526013,"threads_started":2}
I20260806 20:15:44.541615 32445 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:44.542376 31388 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:44.544361 32458 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:44.544948 32458 raft_consensus.cc:686] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:44.546144 30832 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:44.546504 32271 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:15:44.546154 32458 consensus_queue.cc:237] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.532226 32141 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.552204 30676 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:44.552917 32445 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:15:44.545015 32471 log.cc:924] Time spent T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260806 20:15:44.544353 30709 raft_consensus.cc:1261] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:44.495174 30703 rpcz_store.cc:276] 0806 20:15:43.191278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.426426 (+1235148us) service_pool.cc:227] Handling call
0806 20:15:44.442426 (+ 16000us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.446436 (+ 4010us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.458073 (+ 11637us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:44.563318 30785 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:44.564694 30835 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:44.569697 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 65653.3us
I20260806 20:15:44.532238 32467 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260806 20:15:44.558714 32445 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:44.574309 32013 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:44.576123 30835 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:44.578059 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 153129us
I20260806 20:15:44.583236 32454 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:44.583426 32474 log.cc:924] Time spent T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260806 20:15:44.585667 31389 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:44.583993 32078 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:44.580381 31957 consensus_queue.cc:1035] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260806 20:15:44.578945 32149 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260806 20:15:44.589582 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.571074 32271 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260806 20:15:44.586126 30756 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:44.587729 32444 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:44.618263 32423 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:44.618770 32443 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:44.621253 30835 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:44.622164 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 76295.8us
I20260806 20:15:44.627753 32423 raft_consensus.cc:686] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:44.633131 30722 tablet_copy_service.cc:342] P 42cabca48db7403b8523a3382a046043: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-aa7cfe759ca64ca89a0529c2d3da5bf7 received from {username='slave'} at 127.0.0.1:41700
I20260806 20:15:44.634009 30722 tablet_copy_service.cc:433] P 42cabca48db7403b8523a3382a046043: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-aa7cfe759ca64ca89a0529c2d3da5bf7 on tablet aa7cfe759ca64ca89a0529c2d3da5bf7 with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:44.634368 32423 consensus_queue.cc:237] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.638459 32445 raft_consensus.cc:979] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: : Instructing follower 58f2f6256ee14cb38a1dbcbf5bb590a0 to start an election
I20260806 20:15:44.639150 32430 raft_consensus.cc:1067] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Signalling peer 58f2f6256ee14cb38a1dbcbf5bb590a0 to start an election
I20260806 20:15:44.623837 32234 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:44.644637 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1480) took 1270 ms. Trace:
I20260806 20:15:44.622985 32444 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:44.625475 32443 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:44.627729 32370 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:44.646744 30791 rpcz_store.cc:276] 0806 20:15:43.374531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.222994 (+848463us) service_pool.cc:227] Handling call
0806 20:15:44.269570 (+ 46576us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:44.321032 (+ 51462us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:44.321066 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:44.334734 (+ 13668us) log.cc:842] Serialized 26 byte log entry
0806 20:15:44.448671 (+113937us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:44.448707 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.448774 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:44.644363 (+195589us) raft_consensus.cc:1671] finished
0806 20:15:44.644391 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:44.644539 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12824,"raft.queue_time_us":25459,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":141,"thread_start_us":96632,"threads_started":1,"wal-append.queue_time_us":223810}
I20260806 20:15:44.652887 30835 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.654065 32282 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:44.654515 32282 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:44.654901 32282 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:44.627813 32426 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:44.656766 30673 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:44.642309 30804 tablet_copy_service.cc:342] P d4181d4d907f477f9feddc44ba3b9207: Request end of tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-5ba1969a3911423e8e51fc4f2962515d received from {username='slave'} at 127.0.0.1:53732
I20260806 20:15:44.650429 32444 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:44.675065 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 93) took 3425 ms. Trace:
I20260806 20:15:44.653779 32370 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:44.642149 32345 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:44.652367 32443 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:44.650135 30703 raft_consensus.cc:1261] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:44.649190 32078 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:44.684984 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 53116.1us
I20260806 20:15:44.677078 30835 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.671527 32422 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.678733 32370 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:44.691237 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 112010us
W20260806 20:15:44.691609 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 59448us
I20260806 20:15:44.679364 32426 raft_consensus.cc:686] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:44.693887 30832 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:44.688349 32430 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:44.692392 30804 tablet_copy_service.cc:433] P d4181d4d907f477f9feddc44ba3b9207: ending tablet copy session 58f2f6256ee14cb38a1dbcbf5bb590a0-5ba1969a3911423e8e51fc4f2962515d on tablet 5ba1969a3911423e8e51fc4f2962515d with peer 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:44.713891 30786 raft_consensus.cc:1261] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:44.715287 32274 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.716683 32282 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.717746 30756 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.720268 32428 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:44.721534 32238 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:44.721750 31415 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e"
dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
from {username='slave'} at 127.0.0.1:49046
W20260806 20:15:44.724643 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 55716us
I20260806 20:15:44.735714 30676 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:44.727617 32234 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:44.750000 32234 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:44.750962 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 60207.3us
I20260806 20:15:44.743005 31415 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:15:44.753755 32244 consensus_queue.cc:1035] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:44.754729 32282 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:44.753925 31415 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:44.729993 32430 raft_consensus.cc:686] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:44.773339 30832 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:44.726466 32467 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260806 20:15:44.765897 32234 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:44.786907 30756 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.768254 32445 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:44.766414 32238 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.785943 32271 consensus_queue.cc:1035] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260806 20:15:44.775590 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:44.716643 32426 consensus_queue.cc:237] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.790428 32274 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.774489 32345 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:15:44.795958 32482 log.cc:924] Time spent T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
W20260806 20:15:44.797293 32485 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20260806 20:15:44.790371 32430 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.799775 32468 raft_consensus.cc:2741] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:15:44.761541 32242 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:44.805361 31415 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.834936 32282 consensus_queue.cc:1046] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:44.678210 30713 rpcz_store.cc:276] 0806 20:15:41.249326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.273405 (+3024079us) service_pool.cc:227] Handling call
0806 20:15:44.299157 (+ 25752us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:44.331974 (+ 32817us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:15:44.421221 (+ 89247us) log.cc:842] Serialized 14 byte log entry
0806 20:15:44.674766 (+253545us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:44.674814 (+ 48us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:15:44.674852 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.674911 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:44.675012 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":127879,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":5839,"thread_start_us":310445,"threads_started":2}
I20260806 20:15:44.820020 30756 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.788592 32283 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:15:44.818470 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 56203.4us
I20260806 20:15:44.824080 30835 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.801822 32445 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:44.837579 30756 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:44.820664 32244 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:15:44.838080 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 128941us
I20260806 20:15:44.820703 32345 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:44.841300 32445 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:44.842710 31415 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.843765 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:44.850762 30756 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.851018 30713 raft_consensus.cc:3032] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:44.851372 30713 raft_consensus.cc:729] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:44.774745 32428 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.845896 32078 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:44.846341 32443 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.840595 32103 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:44.859009 30756 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.860880 30756 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:44.861536 32388 raft_consensus.cc:2741] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 20:15:44.850659 32345 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:44.845819 32238 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.855686 31414 raft_consensus.cc:2396] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:44.854636 32141 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:44.871976 30756 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:44.859654 32274 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:44.858814 31417 raft_consensus.cc:1226] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:44.862620 32103 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:44.888238 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 54199us
I20260806 20:15:44.885396 30713 consensus_queue.cc:260] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:44.890239 30713 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:44.874276 32141 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:44.880798 32388 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:15:44.903831 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 79868.7us
I20260806 20:15:44.884686 32274 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:44.880324 32244 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260806 20:15:44.887125 30786 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 20:15:44.893687 32487 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260806 20:15:44.888828 32103 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:44.924441 32282 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:44.902478 32141 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:44.931605 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 43753.5us
I20260806 20:15:44.891597 32078 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
I20260806 20:15:44.910290 32370 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.915630 32274 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:44.914453 32468 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:44.893832 32283 raft_consensus.cc:686] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:44.945847 32468 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:44.946429 32468 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:44.946775 32283 consensus_queue.cc:237] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.951437 32370 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:44.952529 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 99575.4us
I20260806 20:15:44.954818 32274 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:44.956561 30676 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:44.975001 32508 log.cc:924] Time spent T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260806 20:15:44.976366 32222 consensus_queue.cc:1035] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:44.993041 30713 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
W20260806 20:15:44.995332 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 103136us
I20260806 20:15:44.998353 31388 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:44.979442 32388 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:45.005558 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 89711.4us
W20260806 20:15:45.004122 32488 log.cc:924] Time spent T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.162s user 0.001s sys 0.002s
I20260806 20:15:44.981932 32422 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:44.994837 32282 raft_consensus.cc:686] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:45.020004 32509 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.021765 32500 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.022358 32500 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.024434 32500 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:45.025279 32480 log.cc:924] Time spent T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.449s user 0.000s sys 0.003s
I20260806 20:15:45.025606 32511 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260806 20:15:45.020637 32013 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:45.027746 32013 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:44.999846 32430 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:45.032194 32491 log.cc:924] Time spent T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.190s user 0.001s sys 0.000s
I20260806 20:15:45.032740 32428 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.000591 32443 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:45.027516 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1485) took 1429 ms. Trace:
I20260806 20:15:45.007380 32283 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:45.027762 32502 log.cc:924] Time spent T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20260806 20:15:45.044420 32274 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.044209 32422 consensus_queue.cc:1035] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:45.041839 32443 raft_consensus.cc:2932] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:45.020790 32509 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.045855 32510 consensus_queue.cc:1035] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:45.047829 32141 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:45.047240 32013 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.049427 32282 consensus_queue.cc:237] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.018452 32388 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.046986 30676 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:45.055302 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:45.055490 32468 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.034569 30789 rpcz_store.cc:276] 0806 20:15:43.598526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:44.240718 (+642192us) service_pool.cc:227] Handling call
0806 20:15:44.254586 (+ 13868us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:44.303298 (+ 48712us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:44.303335 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:44.312766 (+ 9431us) log.cc:842] Serialized 26 byte log entry
0806 20:15:44.436876 (+124110us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:44.436921 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:44.436984 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:44.913984 (+477000us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:15:45.027162 (+113178us) raft_consensus.cc:1671] finished
0806 20:15:45.027182 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:45.027452 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2109,"raft.queue_time_us":19008,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":193,"spinlock_wait_cycles":11326720,"thread_start_us":86493,"threads_started":1,"wal-append.queue_time_us":219890}
I20260806 20:15:45.058133 32426 raft_consensus.cc:3032] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:45.061553 32244 consensus_queue.cc:1035] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:45.066947 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.047139 32103 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:45.068467 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 163835us
I20260806 20:15:45.081780 32426 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:45.084875 32013 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.091698 32370 raft_consensus.cc:2741] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:15:45.076022 32322 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:15:45.085153 32468 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:45.104952 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 49848.7us
I20260806 20:15:45.067873 32234 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:45.110445 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 117895us
W20260806 20:15:45.111099 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 64223.4us
I20260806 20:15:45.113416 30756 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.083160 32283 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:45.150748 32509 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:45.124539 32322 raft_consensus.cc:686] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:45.153903 32322 consensus_queue.cc:237] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:45.154711 32445 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.156709 32445 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.150398 32444 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:45.128065 32511 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:45.161170 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:45.170444 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.172539 32524 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.177204 30713 raft_consensus.cc:1261] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:45.108055 32426 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.159072 32370 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.173224 32524 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:45.180505 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 52915.2us
W20260806 20:15:45.180445 31389 leader_election.cc:334] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.594s (SENT)
I20260806 20:15:45.181157 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.193655 30712 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.194025 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.196825 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:45.197806 30676 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:45.167398 32511 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:45.198478 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.199043 32103 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.199534 32149 raft_consensus.cc:2726] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:45.199882 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:45.200383 31389 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:45.204543 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.204886 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.174193 32238 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:45.229961 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 116551us
I20260806 20:15:45.179446 32445 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:45.233108 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.171283 32013 consensus_queue.cc:1035] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260806 20:15:45.192945 32141 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:45.219856 32511 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:45.242136 30707 raft_consensus.cc:1261] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20260806 20:15:45.243920 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 44122.1us
I20260806 20:15:45.241241 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.245177 30791 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:45.249652 30703 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:45.250418 30784 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:45.250874 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.251617 30703 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:45.252063 32444 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:45.254220 32274 consensus_queue.cc:1035] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:45.246479 30707 pending_rounds.cc:76] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 1
I20260806 20:15:45.198939 32370 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.234440 32430 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:45.267470 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:45.276388 30789 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.263301 30707 pending_rounds.cc:97] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 20:15:45.276516 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.233567 32445 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:45.283583 30791 raft_consensus.cc:1226] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:45.284500 30707 raft_consensus.cc:2866] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:45.279486 30676 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:45.247704 32388 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.258816 32234 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:45.269701 32426 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:45.305737 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.230119 32521 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.123s
W20260806 20:15:45.309799 32528 log.cc:924] Time spent T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260806 20:15:45.310331 32522 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:45.310791 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 130409us
I20260806 20:15:45.310933 32522 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.312448 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.313062 30713 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:45.313539 32513 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:45.314415 32524 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:45.315084 32370 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.275779 30709 raft_consensus.cc:1261] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:45.324762 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.284916 32445 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.313163 32522 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:45.325555 30706 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:45.316704 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.330787 32283 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:45.327628 32422 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
W20260806 20:15:45.330960 32537 log.cc:924] Time spent T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20260806 20:15:45.331233 32078 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.317616 32510 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.288093 32430 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:45.266222 32238 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.305729 32444 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.305646 32244 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.336616 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.325835 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:45.347407 30707 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.346688 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:45.351876 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 79868.9us
I20260806 20:15:45.352406 30756 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:45.352061 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.355576 30710 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:45.359234 30711 raft_consensus.cc:1261] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260806 20:15:45.364813 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 53194.6us
I20260806 20:15:45.342950 32430 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:45.365604 30676 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:45.367272 30710 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:45.371724 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.372632 30707 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:45.357339 30784 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.346663 32345 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.375638 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:45.376258 30790 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.377339 32345 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.377995 32238 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.345616 32388 consensus_queue.cc:1035] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260806 20:15:45.379699 30786 raft_consensus.cc:1261] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 20:15:45.380061 30786 pending_rounds.cc:76] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207: Aborting all ops after (but not including) 2
I20260806 20:15:45.380293 30786 pending_rounds.cc:97] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 20:15:45.380530 30786 raft_consensus.cc:2866] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:45.381557 32244 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:45.384599 30756 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.385780 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.386231 32244 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:45.363719 32141 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:45.383611 32078 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:45.355091 30709 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:45.388319 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.388850 32078 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:45.387090 32388 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:45.389670 32388 raft_consensus.cc:686] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:45.390563 32388 consensus_queue.cc:237] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.396800 30786 raft_consensus.cc:1261] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:45.398272 30867 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:45.399426 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.403115 30712 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:45.404647 30706 raft_consensus.cc:2396] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:45.387014 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.405526 30710 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:45.405946 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.406752 30789 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:45.407717 31389 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:45.408176 32533 log.cc:924] Time spent T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20260806 20:15:45.408628 32518 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.408810 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.409247 32518 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.409677 32545 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:45.410522 30707 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:45.396816 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.411324 30756 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.414410 30713 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:45.416163 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.388872 30709 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:45.417155 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.415275 30756 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.417927 30861 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:45.418098 32388 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:45.344254 32510 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:45.419715 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 54204.6us
I20260806 20:15:45.419648 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.420296 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.374398 32430 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:45.346846 32444 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.420894 30705 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.421002 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.418699 32388 raft_consensus.cc:686] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:45.418741 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.423265 30871 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:45.423123 32388 consensus_queue.cc:237] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.424140 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.424919 30871 raft_consensus.cc:2396] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:45.426131 30870 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:45.416234 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.431664 30712 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.420706 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.440822 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.441939 30713 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.440668 32103 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:45.442977 32103 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:45.443079 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.418223 30708 raft_consensus.cc:1261] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:45.444317 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.445044 30866 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:45.446451 30676 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:45.410068 32149 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:45.453387 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.454299 30707 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.450281 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:45.457650 30706 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.410095 32545 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:45.464913 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.448217 32103 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.481431 30790 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:45.445653 32370 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:45.491501 32542 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260806 20:15:45.444291 32458 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 20:15:45.441315 32521 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
I20260806 20:15:45.512414 30709 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:45.494040 30786 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:45.495265 32444 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:45.498167 30791 raft_consensus.cc:1226] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
W20260806 20:15:45.498795 32538 log.cc:924] Time spent T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20260806 20:15:45.517467 30790 raft_consensus.cc:1226] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:45.518595 30866 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:45.501322 30703 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:45.483039 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.502059 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:15:45.506906 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 60562us
I20260806 20:15:45.517644 30756 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.509111 30867 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:45.512411 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.464210 32428 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:45.464454 32545 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:45.515174 30707 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:45.482846 32468 consensus_queue.cc:1035] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
I20260806 20:15:45.464267 32274 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:45.469058 32141 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.500506 32510 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:45.521737 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.524398 30713 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:45.531311 30712 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:45.507463 32518 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:45.534900 30756 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.536825 30712 raft_consensus.cc:1226] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260806 20:15:45.568331 30710 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:45.570727 30676 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:45.512197 32370 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:45.562284 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:45.526124 30861 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:45.572029 30710 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:15:45.595536 30871 raft_consensus.cc:1261] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260806 20:15:45.606860 30866 raft_consensus.cc:1261] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:45.519606 32013 consensus_queue.cc:1035] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260806 20:15:45.572382 30756 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.595721 30790 raft_consensus.cc:1226] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:45.539361 32388 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:15:45.663558 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 103959us
I20260806 20:15:45.607364 31422 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:45.547575 30784 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:45.606715 32550 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.650161 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.639752 30706 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:45.669339 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:45.669945 30706 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:45.659664 30871 pending_rounds.cc:76] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 2
W20260806 20:15:45.665628 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 93306.3us
I20260806 20:15:45.585016 32422 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:45.608741 30705 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:45.605623 32518 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.674044 30756 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.677515 30871 pending_rounds.cc:97] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260806 20:15:45.656483 32149 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:45.596381 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:45.554059 32274 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:45.688035 31420 raft_consensus.cc:2396] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:45.688651 32238 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:15:45.689742 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.690804 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:45.691435 30790 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:15:45.692489 30757 leader_election.cc:334] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.528s (SENT)
I20260806 20:15:45.692966 30757 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:45.684691 30871 raft_consensus.cc:2866] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:45.675062 32345 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:45.554631 32141 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.759541 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.678088 32550 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.678898 30710 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:15:45.756731 30756 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.762349 30867 raft_consensus.cc:1261] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:45.763815 30706 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:45.599790 32511 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.770103 30706 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:15:45.695478 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:45.769572 32444 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.774495 32542 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.022s
I20260806 20:15:45.780283 32388 raft_consensus.cc:686] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:45.764587 32149 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:45.776115 32013 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:15:45.803884 32552 log.cc:924] Time spent T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20260806 20:15:45.798651 30713 raft_consensus.cc:1261] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:45.802122 30792 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:45.810755 30710 raft_consensus.cc:1226] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 20:15:45.802716 30787 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:45.803054 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.809033 30790 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:45.865176 30785 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:45.797260 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:45.804497 32556 log.cc:924] Time spent T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20260806 20:15:45.867923 30788 raft_consensus.cc:1261] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:15:45.868568 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.869112 30788 raft_consensus.cc:1261] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:45.870414 32550 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.870664 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.871851 32511 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.850795 30868 raft_consensus.cc:1261] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:45.872167 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.872188 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.767554 31418 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:45.875002 30832 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:15:45.805169 32557 log.cc:924] Time spent T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260806 20:15:45.849874 32513 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:15:45.800271 32534 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:45.795131 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 127906us
I20260806 20:15:45.801564 32468 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:45.876086 32534 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.876549 32468 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:45.877043 32468 raft_consensus.cc:686] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:45.877662 32510 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:45.878275 32534 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.873097 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:45.844724 32426 consensus_meta.cc:287] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.519s user 0.013s sys 0.000s
I20260806 20:15:45.850306 32454 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:45.878859 32534 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.878151 32506 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:45.879220 32454 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:45.851305 32370 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:45.879885 32506 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:45.758023 32518 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.771978 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:45.880331 32506 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:45.879328 32534 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.776333 32103 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.884433 32103 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.885234 30789 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:45.879431 30786 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:45.886665 32530 consensus_queue.cc:1046] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260806 20:15:45.887253 32518 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:45.865584 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.867503 32388 consensus_queue.cc:237] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.871954 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.896256 30870 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:45.771126 32078 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.883121 32468 consensus_queue.cc:237] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:45.805771 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 122585us
I20260806 20:15:45.802186 30703 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:45.803141 32500 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:45.956887 32500 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:45.957664 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:45.864154 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 199890us
I20260806 20:15:45.959242 30703 raft_consensus.cc:2379] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:45.959368 32500 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.960745 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.961784 30703 raft_consensus.cc:2396] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:45.962678 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.963380 30703 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:45.964430 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:45.887795 32518 raft_consensus.cc:3032] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:45.847090 32422 raft_consensus.cc:686] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:45.974941 30790 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:45.968986 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:45.870545 32141 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:45.977068 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.979521 30713 raft_consensus.cc:1226] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:45.774921 32271 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.165s
I20260806 20:15:45.986618 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.973928 30835 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.989578 30706 raft_consensus.cc:1261] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:45.985577 30756 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:45.990669 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.000536 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:45.807511 32444 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.003043 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:46.012570 30713 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:46.004055 30835 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:46.005731 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:46.006459 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.009642 30676 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:45.967598 32426 consensus_queue.cc:1035] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260806 20:15:46.019131 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:45.973543 32518 raft_consensus.cc:729] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:45.972798 30783 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:46.015064 32141 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:46.010232 32536 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:46.051084 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.026728 30863 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:46.027119 32510 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:46.040486 32141 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:46.029227 32345 raft_consensus.cc:686] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:45.965309 30703 raft_consensus.cc:2396] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 5 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:46.063899 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.076386 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.077522 32422 consensus_queue.cc:237] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.077060 30871 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:46.079698 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.039889 32370 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:46.080394 32370 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:46.080790 32370 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:46.068143 32141 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:46.082195 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.083410 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.088541 32370 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.064093 32530 consensus_queue.cc:1046] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20260806 20:15:46.092756 32370 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:46.027616 32565 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.066s
I20260806 20:15:46.078001 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.093532 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.094092 30703 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:46.094255 30786 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:46.094797 32510 raft_consensus.cc:686] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:46.095321 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.063104 32536 raft_consensus.cc:686] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:46.069509 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.072718 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.073529 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.073817 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.098851 30788 raft_consensus.cc:2379] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:46.069509 30783 raft_consensus.cc:1226] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:46.079205 32345 consensus_queue.cc:237] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:46.095844 32510 consensus_queue.cc:237] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.103945 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.026412 32443 consensus_queue.cc:1046] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260806 20:15:46.088537 32506 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.111243 30756 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:46.113312 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.096175 30786 raft_consensus.cc:2379] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:46.066874 32534 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:46.098839 32445 consensus_meta.cc:287] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.763s user 0.012s sys 0.002s
I20260806 20:15:46.140470 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.049242 30713 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:45.982190 32078 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.076995 32518 consensus_queue.cc:260] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.202188 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:46.203431 30757 leader_election.cc:334] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.559s (SENT)
I20260806 20:15:46.249670 30756 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:46.249492 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.218813 30676 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:46.143710 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:46.258010 30869 raft_consensus.cc:2396] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:46.257462 30790 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:46.258893 32078 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:46.259371 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.260170 32518 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:46.141376 30784 raft_consensus.cc:2354] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4.
I20260806 20:15:46.261788 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:46.190271 32370 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:46.262193 30784 raft_consensus.cc:1261] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 20:15:46.176658 32430 consensus_meta.cc:287] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.755s user 0.009s sys 0.004s
I20260806 20:15:46.263306 32430 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.264524 30707 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:46.264653 30863 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:46.264835 30788 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:46.254056 30713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7"
dest_uuid: "42cabca48db7403b8523a3382a046043"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:38516
I20260806 20:15:46.209014 32506 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:46.252384 30787 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:46.262805 32370 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:46.175705 32545 consensus_meta.cc:287] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.616s user 0.009s sys 0.001s
W20260806 20:15:46.207412 32543 log.cc:924] Time spent T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.814s user 0.014s sys 0.016s
W20260806 20:15:46.204316 32547 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.866s user 0.010s sys 0.007s
I20260806 20:15:46.270079 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.204174 30703 raft_consensus.cc:1226] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:46.260866 32141 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.202814 32534 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:46.202034 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:15:46.278324 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 59626us
I20260806 20:15:46.285972 30713 raft_consensus.cc:595] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:46.286943 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 83558.9us
I20260806 20:15:46.287561 30713 raft_consensus.cc:603] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:46.203127 32454 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:46.275839 30867 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:46.254070 32445 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.286072 32565 raft_consensus.cc:2741] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:15:46.286203 32013 raft_consensus.cc:686] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:46.289148 32534 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:46.204211 32536 consensus_queue.cc:237] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.286346 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:46.317288 30712 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.014s sys 0.077s
W20260806 20:15:46.335287 30712 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.015s sys 0.077s
I20260806 20:15:46.328079 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.276974 32545 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.281085 32430 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:46.338121 30867 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:46.317593 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:46.342500 30676 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:46.323426 32444 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:46.332217 32534 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:46.348963 32565 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:46.351919 31415 raft_consensus.cc:2379] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:46.343464 32568 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:46.354473 32013 consensus_queue.cc:237] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.355716 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 119) took 1097 ms. Trace:
I20260806 20:15:46.356138 30707 rpcz_store.cc:276] 0806 20:15:45.258488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.522699 (+264211us) service_pool.cc:227] Handling call
0806 20:15:45.522968 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:46.264964 (+741996us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:46.355418 (+ 90454us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b5400175054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:46.355645 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1705071488}
I20260806 20:15:46.358029 32513 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:15:46.358917 32513 raft_consensus.cc:3032] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:46.359238 32513 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:46.360035 32513 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.361356 32513 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:46.364321 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:46.365149 32565 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:46.365626 30676 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:46.366412 32565 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:46.366812 30676 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:46.367908 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.369261 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.370451 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.371588 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.372434 30867 raft_consensus.cc:2396] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:46.373901 32534 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.378539 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.379271 30864 raft_consensus.cc:2362] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5. Re-sending same reply.
I20260806 20:15:46.379467 32534 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:46.380514 30864 raft_consensus.cc:1226] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:46.382431 32141 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:46.384939 32574 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:46.336934 32454 consensus_queue.cc:1046] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260806 20:15:46.385560 32574 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.386268 32370 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:46.388278 30867 raft_consensus.cc:1261] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:46.373330 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:46.372314 32559 log.cc:924] Time spent T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.527s user 0.000s sys 0.005s
I20260806 20:15:46.390492 32568 raft_consensus.cc:686] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:15:46.416093 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 49402.9us
I20260806 20:15:46.416863 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:46.417143 31389 leader_election.cc:334] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.600s (SENT)
I20260806 20:15:46.404736 32370 raft_consensus.cc:2741] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 20:15:46.405928 32574 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:46.417608 31389 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:46.433637 30710 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.000s sys 0.016s
W20260806 20:15:46.434027 30710 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s user 0.001s sys 0.017s
I20260806 20:15:46.413887 30703 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:46.413215 32545 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:46.442665 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.449781 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:46.453472 30676 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:46.459266 30676 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:46.439553 32430 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:46.490818 32445 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:46.457131 32534 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:46.500567 32573 log.cc:924] Time spent T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260806 20:15:46.543668 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1114) took 1158 ms. Trace:
W20260806 20:15:46.501590 32575 log.cc:924] Time spent T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.160s user 0.001s sys 0.001s
I20260806 20:15:46.544117 30705 rpcz_store.cc:276] 0806 20:15:45.384938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.697916 (+312978us) service_pool.cc:227] Handling call
0806 20:15:45.845981 (+148065us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:45.850245 (+ 4264us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:45.850281 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:45.850765 (+ 484us) log.cc:842] Serialized 26 byte log entry
0806 20:15:46.115539 (+264774us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:46.127829 (+ 12290us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:46.139476 (+ 11647us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:46.543393 (+403917us) raft_consensus.cc:1671] finished
0806 20:15:46.543414 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:46.543593 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":104,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"thread_start_us":206218,"threads_started":1,"wal-append.queue_time_us":472558}
I20260806 20:15:46.502673 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1395) took 1272 ms. Trace:
I20260806 20:15:46.546312 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.546620 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1112) took 1219 ms. Trace:
I20260806 20:15:46.547022 30709 rpcz_store.cc:276] 0806 20:15:45.326980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.569629 (+242649us) service_pool.cc:227] Handling call
0806 20:15:45.610643 (+ 41014us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:45.810396 (+199753us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:45.810431 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:45.810895 (+ 464us) log.cc:842] Serialized 26 byte log entry
0806 20:15:46.114160 (+303265us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:46.114196 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:46.114257 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:46.546398 (+432141us) raft_consensus.cc:1671] finished
0806 20:15:46.546418 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:46.546560 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9280,"raft.queue_time_us":34126,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":137,"thread_start_us":373569,"threads_started":2,"wal-append.queue_time_us":485997}
I20260806 20:15:46.548821 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.478564 32238 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:46.487051 32565 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:46.509697 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1111) took 1194 ms. Trace:
I20260806 20:15:46.545924 30708 rpcz_store.cc:276] 0806 20:15:45.229981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.489089 (+259108us) service_pool.cc:227] Handling call
0806 20:15:45.489524 (+ 435us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:45.489801 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:45.489829 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:45.490301 (+ 472us) log.cc:842] Serialized 26 byte log entry
0806 20:15:45.649886 (+159585us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:45.656349 (+ 6463us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:45.661324 (+ 4975us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:46.371744 (+710420us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:15:46.502316 (+130572us) raft_consensus.cc:1671] finished
0806 20:15:46.502349 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:46.502606 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7389952,"thread_start_us":126665,"threads_started":1,"wal-append.queue_time_us":211194}
W20260806 20:15:46.551049 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 91874us
I20260806 20:15:46.551002 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.503582 30866 raft_consensus.cc:1261] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:46.513000 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.513885 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.448933 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.518134 32568 consensus_queue.cc:237] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.564541 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:46.555923 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.565816 30864 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:46.566921 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.567618 30864 raft_consensus.cc:2362] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2. Re-sending same reply.
I20260806 20:15:46.568692 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1095) took 1122 ms. Trace:
I20260806 20:15:46.549999 30712 rpcz_store.cc:276] 0806 20:15:45.315093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.565720 (+250627us) service_pool.cc:227] Handling call
0806 20:15:45.644752 (+ 79032us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:45.694035 (+ 49283us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:45.694061 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:45.756837 (+ 62776us) log.cc:842] Serialized 26 byte log entry
0806 20:15:46.340162 (+583325us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:46.340207 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:46.340279 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:46.509475 (+169196us) raft_consensus.cc:1671] finished
0806 20:15:46.509495 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:46.509633 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18331,"raft.queue_time_us":29633,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":8778,"thread_start_us":554660,"threads_started":2,"wal-append.queue_time_us":684265}
W20260806 20:15:46.523579 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 106450us
W20260806 20:15:46.524518 30757 leader_election.cc:334] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.848s (SENT)
I20260806 20:15:46.494215 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:46.488828 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 463975us
I20260806 20:15:46.457909 32545 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:46.547858 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.511747 32271 raft_consensus.cc:3032] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:46.553400 30676 leader_election.cc:398] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4.
I20260806 20:15:46.569084 30870 rpcz_store.cc:276] 0806 20:15:45.445891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.483093 (+ 37202us) service_pool.cc:227] Handling call
0806 20:15:46.568478 (+1085385us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54002a0654. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:46.568594 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":827562880}
I20260806 20:15:46.586400 30676 leader_election.cc:401] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:15:46.554246 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.555831 32274 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:46.581073 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.618306 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:46.618490 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.567341 32426 raft_consensus.cc:3032] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:46.585165 30835 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:46.566793 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:46.589352 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 64882us
I20260806 20:15:46.549435 32238 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:46.616760 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.568711 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.581962 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.618979 30709 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:46.632042 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.633885 30870 raft_consensus.cc:3032] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:46.640870 30870 raft_consensus.cc:729] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:46.635039 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:46.644001 32507 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:46.644595 32507 raft_consensus.cc:686] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:46.641741 30870 consensus_queue.cc:260] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.645941 30870 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:46.644052 30787 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:46.646515 30866 raft_consensus.cc:3032] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:46.646909 30866 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:46.647833 32565 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4.
I20260806 20:15:46.647634 30866 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.648593 30866 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:46.553791 32576 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.128s
I20260806 20:15:46.650606 32507 consensus_queue.cc:237] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.657271 32545 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:46.657889 32574 consensus_queue.cc:1035] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:46.658159 32513 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260806 20:15:46.636211 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 51145.4us
I20260806 20:15:46.636708 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.662205 30835 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:46.668004 30709 raft_consensus.cc:2396] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:46.586171 32271 raft_consensus.cc:729] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:46.666721 30835 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:46.675259 30787 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:46.676805 30866 raft_consensus.cc:1261] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 2 index: 2. (term mismatch)
I20260806 20:15:46.677219 30866 pending_rounds.cc:76] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 1
I20260806 20:15:46.666705 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1115) took 1226 ms. Trace:
W20260806 20:15:46.679013 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 125772us
I20260806 20:15:46.683199 30870 raft_consensus.cc:1261] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:46.555284 32534 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.637329 32238 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:46.583797 32510 consensus_queue.cc:1046] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.082s
I20260806 20:15:46.623692 32426 raft_consensus.cc:729] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:46.684176 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:46.684880 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.686087 30835 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:46.687011 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:46.587440 32078 consensus_queue.cc:1046] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.080s
I20260806 20:15:46.693610 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:46.683876 30708 raft_consensus.cc:2379] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:46.710312 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:46.711024 30789 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:46.711378 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.707870 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.712568 30865 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:46.712705 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:46.713989 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.714522 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:46.714648 30708 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:46.715286 30789 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:46.715639 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.699117 30709 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:46.657835 32545 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.700013 32426 consensus_queue.cc:260] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.716840 32426 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:46.718052 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.718043 32545 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:46.720456 32578 consensus_queue.cc:1035] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:46.703934 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:46.671226 32518 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:46.707093 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.707934 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:15:46.708315 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:46.722447 32518 raft_consensus.cc:686] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:46.722919 30712 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:46.709038 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:46.723419 30835 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:46.725052 30787 raft_consensus.cc:595] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:46.725761 30787 raft_consensus.cc:603] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:46.722484 30867 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:46.670521 32430 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:46.691176 32443 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:46.670192 32569 consensus_queue.cc:1035] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.088s
I20260806 20:15:46.694964 32370 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:46.677438 30866 pending_rounds.cc:97] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260806 20:15:46.769898 30866 raft_consensus.cc:2866] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:46.758920 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:46.770834 30870 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:46.690454 32513 consensus_queue.cc:1046] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.074s
I20260806 20:15:46.694685 32506 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260806 20:15:46.694149 32445 consensus_queue.cc:1035] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260806 20:15:46.760772 32443 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:46.810037 32468 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s user 0.008s sys 0.014s
W20260806 20:15:46.810510 32468 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.855s user 0.008s sys 0.014s
I20260806 20:15:46.812319 32468 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 20:15:46.805934 30790 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.033s sys 0.026s
W20260806 20:15:46.813792 30790 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.524s user 0.033s sys 0.026s
I20260806 20:15:46.843076 30791 raft_consensus.cc:1226] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:46.848286 30712 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:46.765802 32580 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:46.851104 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:46.853613 30712 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:46.854589 31388 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:46.856662 30870 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:46.765342 32430 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.868335 30867 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:46.838155 32443 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:15:46.842885 32515 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.466s user 0.007s sys 0.020s
I20260806 20:15:46.800875 32370 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:46.905021 30711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.011s sys 0.016s
W20260806 20:15:46.903057 30785 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.010s sys 0.009s
W20260806 20:15:46.909396 30711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s user 0.011s sys 0.016s
W20260806 20:15:46.909655 30785 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s user 0.010s sys 0.010s
I20260806 20:15:46.927362 30792 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:15:46.928118 30792 raft_consensus.cc:595] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:46.928447 30792 raft_consensus.cc:603] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:46.863507 32430 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:46.864414 32545 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:46.870013 32578 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:46.904719 32515 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.528s user 0.008s sys 0.020s
I20260806 20:15:46.766309 32271 consensus_queue.cc:260] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.907469 31415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 317) took 1251 ms. Trace:
I20260806 20:15:46.963864 30712 raft_consensus.cc:1261] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:46.934628 32578 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:46.948627 32430 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:46.960832 31415 rpcz_store.cc:276] 0806 20:15:45.651356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.661008 (+ 9652us) service_pool.cc:227] Handling call
0806 20:15:46.857422 (+1196414us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b54004c2cd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:46.871206 (+ 13784us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1251916032}
I20260806 20:15:46.872215 32518 consensus_queue.cc:237] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:46.967581 32578 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:46.946250 32545 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:46.970122 32430 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:46.973356 32534 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:47.008358 30868 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s user 0.020s sys 0.009s
I20260806 20:15:46.988112 30709 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:47.011855 30709 raft_consensus.cc:1226] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260806 20:15:46.991159 32430 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:46.999805 32506 raft_consensus.cc:3032] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 20:15:47.126212 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 453925us
I20260806 20:15:47.134279 32422 consensus_queue.cc:1035] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:47.136868 32422 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.137391 32422 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.139581 32422 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.123806 32506 raft_consensus.cc:729] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:15:47.157569 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 479252us
W20260806 20:15:47.159478 30707 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.010s sys 0.013s
W20260806 20:15:47.159943 30707 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s user 0.010s sys 0.013s
I20260806 20:15:47.162631 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1513) took 1574 ms. Trace:
I20260806 20:15:47.163066 30791 rpcz_store.cc:276] 0806 20:15:45.585199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.585406 (+ 207us) service_pool.cc:227] Handling call
0806 20:15:45.585735 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:46.843503 (+1257768us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.158163 (+314660us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54003b1ad4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:47.159609 (+ 1446us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2891203328}
I20260806 20:15:47.195864 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 98) took 1137 ms. Trace:
W20260806 20:15:47.162158 32141 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.006s sys 0.023s
W20260806 20:15:47.185191 32588 log.cc:924] Time spent T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.194s user 0.000s sys 0.000s
W20260806 20:15:47.202726 32586 log.cc:924] Time spent T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20260806 20:15:47.174540 31414 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 LEARNER]: Advancing to term 3
I20260806 20:15:47.202183 32545 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:47.214044 32141 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s user 0.010s sys 0.027s
I20260806 20:15:47.183835 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1117) took 1305 ms. Trace:
W20260806 20:15:47.214720 32568 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.006s sys 0.026s
I20260806 20:15:47.212385 32370 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:47.147697 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 442984us
I20260806 20:15:47.255771 30676 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:47.251507 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 107591us
I20260806 20:15:47.212651 30785 rpcz_store.cc:276] 0806 20:15:46.058942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.061580 (+ 2638us) service_pool.cc:227] Handling call
0806 20:15:46.089526 (+ 27946us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:46.089862 (+ 336us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:46.089886 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:46.374502 (+284616us) spinlock_profiling.cc:243] Waited 9.22 ms on lock 0x7b2c000002e8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:15:46.375075 (+ 573us) log.cc:842] Serialized 52 byte log entry
0806 20:15:46.911834 (+536759us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:46.912158 (+ 324us) log.cc:842] Serialized 14 byte log entry
0806 20:15:46.932341 (+ 20183us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:46.932748 (+ 407us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:47.195500 (+262752us) raft_consensus.cc:1671] finished
0806 20:15:47.195608 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.195793 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19953,"prepare.queue_time_us":24773,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":2897,"raft.queue_time_us":148602,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":5463,"spinlock_wait_cycles":21194624,"thread_start_us":536355,"threads_started":1,"wal-append.queue_time_us":772101}
W20260806 20:15:47.253907 30674 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.005s sys 0.026s
W20260806 20:15:47.257900 30674 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.005s sys 0.026s
W20260806 20:15:47.223958 30755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.010s sys 0.021s
W20260806 20:15:47.260138 30755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.011s sys 0.021s
I20260806 20:15:47.278708 30757 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:47.260934 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:47.253654 32568 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s user 0.007s sys 0.028s
I20260806 20:15:47.188856 32578 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:47.298445 32598 log.cc:924] Time spent T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260806 20:15:47.301409 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1238) took 1230 ms. Trace:
I20260806 20:15:47.261560 32370 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:47.284364 32078 consensus_queue.cc:1225] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer d4181d4d907f477f9feddc44ba3b9207 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 20:15:47.301826 30790 rpcz_store.cc:276] 0806 20:15:46.070961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.260454 (+189493us) service_pool.cc:227] Handling call
0806 20:15:46.269517 (+ 9063us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:46.269919 (+ 402us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:46.269947 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:46.270434 (+ 487us) log.cc:842] Serialized 26 byte log entry
0806 20:15:46.815860 (+545426us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:46.816247 (+ 387us) log.cc:842] Serialized 14 byte log entry
0806 20:15:46.816424 (+ 177us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:46.816496 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:47.301191 (+484695us) raft_consensus.cc:1671] finished
0806 20:15:47.301213 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.301345 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":231,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":47,"raft.queue_time_us":100996,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":21431,"thread_start_us":525643,"threads_started":1,"wal-append.queue_time_us":690200}
I20260806 20:15:47.294219 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1393) took 2085 ms. Trace:
I20260806 20:15:47.365906 30711 rpcz_store.cc:276] 0806 20:15:45.204414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.448624 (+244210us) service_pool.cc:227] Handling call
0806 20:15:45.472118 (+ 23494us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:45.506757 (+ 34639us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:45.530663 (+ 23906us) log.cc:842] Serialized 14 byte log entry
0806 20:15:46.360114 (+829451us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b180044fa40. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e735345 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb 00007f71512f6063 00007f71512f657b 00007f715167988a
0806 20:15:47.289067 (+928953us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:47.289126 (+ 59us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:47.289168 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.289240 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.289365 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":25568,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":6304,"spinlock_wait_cycles":55125248,"thread_start_us":928581,"threads_started":2,"wal-append.queue_time_us":1570919}
W20260806 20:15:47.286586 30863 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.022s sys 0.036s
W20260806 20:15:47.371141 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 66821.8us
W20260806 20:15:47.372211 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 116501us
I20260806 20:15:47.373346 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.368316 30711 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260806 20:15:47.392474 30757 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:47.396343 30711 raft_consensus.cc:1261] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:47.396667 32607 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:47.397600 30711 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:47.398814 30711 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:47.400091 30711 raft_consensus.cc:1261] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:15:47.257668 30703 rpcz_store.cc:276] 0806 20:15:45.878000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.494761 (+616761us) service_pool.cc:227] Handling call
0806 20:15:46.495130 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:46.495543 (+ 413us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:46.495579 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:46.496061 (+ 482us) log.cc:842] Serialized 26 byte log entry
0806 20:15:46.963070 (+467009us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:46.964622 (+ 1552us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:46.970262 (+ 5640us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:47.183513 (+213251us) raft_consensus.cc:1671] finished
0806 20:15:47.183560 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.183741 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1134,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":460783,"threads_started":1,"wal-append.queue_time_us":685573}
I20260806 20:15:47.402293 32578 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:47.402736 30703 raft_consensus.cc:1261] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:47.402870 32534 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.403388 32534 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:47.405958 32590 log.cc:924] Time spent T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
I20260806 20:15:47.408612 32580 raft_consensus.cc:3032] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:47.409049 32580 raft_consensus.cc:729] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:47.410138 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1116) took 1532 ms. Trace:
I20260806 20:15:47.409950 32580 consensus_queue.cc:260] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.411229 32618 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:47.410558 30707 rpcz_store.cc:276] 0806 20:15:45.877687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.410548 (+532861us) service_pool.cc:227] Handling call
0806 20:15:46.417971 (+ 7423us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:46.436901 (+ 18930us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:46.436931 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:46.444269 (+ 7338us) log.cc:842] Serialized 52 byte log entry
0806 20:15:47.160372 (+716103us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:47.160415 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.160492 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:47.409911 (+249419us) raft_consensus.cc:1671] finished
0806 20:15:47.409936 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.410083 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1348,"raft.queue_time_us":3540,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":11353,"thread_start_us":645512,"threads_started":1,"wal-append.queue_time_us":926038}
I20260806 20:15:47.238790 32506 consensus_queue.cc:260] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.411824 32618 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.412418 32506 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:47.412688 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1113) took 1388 ms. Trace:
I20260806 20:15:47.413057 30868 rpcz_store.cc:276] 0806 20:15:46.024357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.025560 (+ 1203us) service_pool.cc:227] Handling call
0806 20:15:46.054580 (+ 29020us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:46.429066 (+374486us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:46.429093 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:46.429560 (+ 467us) log.cc:842] Serialized 26 byte log entry
0806 20:15:47.009183 (+579623us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:47.009223 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.009302 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:47.412457 (+403155us) raft_consensus.cc:1671] finished
0806 20:15:47.412475 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.412605 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10732,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":8639,"thread_start_us":894903,"threads_started":2,"wal-append.queue_time_us":965674}
I20260806 20:15:47.414546 32618 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:47.417815 30871 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1265) took 1344 ms (client timeout 1608 ms). Trace:
I20260806 20:15:47.417632 32443 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:47.418262 30871 rpcz_store.cc:269] 0806 20:15:46.073896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.079448 (+ 5552us) service_pool.cc:227] Handling call
0806 20:15:47.417613 (+1338165us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b54000843d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:47.417757 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2111373056}
I20260806 20:15:47.420094 32608 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:47.421253 32608 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.421774 32607 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.421713 32608 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.422047 32611 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:47.422307 32607 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.423496 32621 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:47.423940 32621 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:47.424260 32620 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:47.422655 32611 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.424768 31420 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 449) took 1324 ms. Trace:
I20260806 20:15:47.425171 31420 rpcz_store.cc:276] 0806 20:15:46.100420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.104959 (+ 4539us) service_pool.cc:227] Handling call
0806 20:15:47.423993 (+1319034us) spinlock_profiling.cc:243] Waited 32.9 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:47.424225 (+ 232us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:47.424414 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:47.424443 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:47.424487 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:47.424525 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.424584 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.424692 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75555200}
I20260806 20:15:47.424304 32621 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:47.427430 32607 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:47.374008 30863 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.614s user 0.023s sys 0.036s
I20260806 20:15:47.432138 30707 raft_consensus.cc:1261] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260806 20:15:47.371115 32591 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
I20260806 20:15:47.426164 32611 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.401292 30711 raft_consensus.cc:1261] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:47.314985 32513 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:47.427294 32545 raft_consensus.cc:3032] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:47.392000 32422 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.370647 32568 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:47.367748 32585 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:47.442832 30835 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.005s sys 0.005s
W20260806 20:15:47.449929 30835 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.726s user 0.006s sys 0.005s
W20260806 20:15:47.450323 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 726998us
W20260806 20:15:47.443014 32238 consensus_meta.cc:287] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.723s user 0.009s sys 0.002s
I20260806 20:15:47.405396 32534 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.442610 32580 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:47.443548 32513 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:47.451345 30835 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:47.453984 32568 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.444159 32430 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.446463 32545 raft_consensus.cc:729] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:47.461892 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:47.438995 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 279837us
I20260806 20:15:47.442188 32612 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.451939 32422 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:47.444602 30834 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.003s sys 0.007s
I20260806 20:15:47.472028 32513 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:47.532460 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 81176.6us
I20260806 20:15:47.462343 32238 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:47.531800 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1268) took 1457 ms (client timeout 1764 ms). Trace:
I20260806 20:15:47.535511 30676 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:47.536494 30832 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.773s user 0.014s sys 0.010s
W20260806 20:15:47.542132 30832 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.779s user 0.014s sys 0.010s
W20260806 20:15:47.536943 30834 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.750s user 0.003s sys 0.007s
W20260806 20:15:47.536860 30710 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.007s sys 0.012s
W20260806 20:15:47.540244 30869 rpcz_store.cc:269] 0806 20:15:46.074762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.083162 (+ 8400us) service_pool.cc:227] Handling call
0806 20:15:47.524542 (+1441380us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540019d054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:47.531726 (+ 7184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":399731456}
W20260806 20:15:47.546967 30710 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.564s user 0.007s sys 0.012s
I20260806 20:15:47.547693 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1410) took 2156 ms. Trace:
I20260806 20:15:47.509208 32443 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:47.544309 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:46.677772 30706 rpcz_store.cc:276] 0806 20:15:45.440094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.000404 (+560310us) service_pool.cc:227] Handling call
0806 20:15:46.000797 (+ 393us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:46.001157 (+ 360us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:46.001186 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:46.007157 (+ 5971us) log.cc:842] Serialized 52 byte log entry
0806 20:15:46.290836 (+283679us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:46.664925 (+374089us) log.cc:842] Serialized 14 byte log entry
0806 20:15:46.666400 (+ 1475us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:46.666472 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:46.666511 (+ 39us) raft_consensus.cc:1671] finished
0806 20:15:46.666522 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:46.666636 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":398,"raft.queue_time_us":8849,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":10829,"thread_start_us":631502,"threads_started":2,"wal-append.queue_time_us":405754}
I20260806 20:15:47.549842 30710 rpcz_store.cc:276] 0806 20:15:45.391665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.874662 (+482997us) service_pool.cc:227] Handling call
0806 20:15:45.874998 (+ 336us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:45.875197 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:45.875219 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:45.875683 (+ 464us) log.cc:842] Serialized 26 byte log entry
0806 20:15:46.434450 (+558767us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:46.982491 (+548041us) log.cc:842] Serialized 14 byte log entry
0806 20:15:47.547370 (+564879us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.547447 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:47.547491 (+ 44us) raft_consensus.cc:1671] finished
0806 20:15:47.547502 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.547626 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":533668,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":27,"raft.queue_time_us":439170,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"thread_start_us":1619876,"threads_started":3,"wal-append.queue_time_us":794788}
I20260806 20:15:47.537387 32542 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:47.563686 31389 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:47.522043 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1122) took 1553 ms. Trace:
W20260806 20:15:47.550431 32574 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.891s user 0.023s sys 0.061s
I20260806 20:15:47.537676 32565 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:47.578536 32565 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.579387 32609 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:47.579958 32609 raft_consensus.cc:686] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:47.568442 30834 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:47.557798 30676 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:47.514869 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.537489 32613 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:47.543319 32444 raft_consensus.cc:3032] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:47.508289 32585 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.594485 30708 raft_consensus.cc:2379] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:47.580909 32609 consensus_queue.cc:237] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.584103 32078 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.596203 32078 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.596474 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:47.581167 32614 consensus_queue.cc:1046] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260806 20:15:47.595944 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:47.598315 32078 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:47.550320 32638 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:47.550488 32370 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:15:47.604348 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 187512us
W20260806 20:15:47.581704 32574 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.922s user 0.023s sys 0.061s
I20260806 20:15:47.558753 32622 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:47.580863 32565 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.563651 32500 consensus_queue.cc:1046] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260806 20:15:47.592370 32426 consensus_queue.cc:1046] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.025s
I20260806 20:15:47.615767 32510 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:47.617463 32500 raft_consensus.cc:3032] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:47.589272 32613 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:47.618419 32500 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:47.618759 32613 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:47.619366 32500 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.620613 32500 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:47.592242 32444 raft_consensus.cc:729] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:47.622128 32444 consensus_queue.cc:260] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:47.623304 32444 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:47.624363 32422 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.625551 32238 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.627076 32443 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.571062 30705 rpcz_store.cc:276] 0806 20:15:45.968435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.719042 (+750607us) service_pool.cc:227] Handling call
0806 20:15:46.719377 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:46.958317 (+238940us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:46.962774 (+ 4457us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:46.964992 (+ 2218us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:46.966906 (+ 1914us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.521745 (+554839us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b54002b07d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:47.521803 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.521974 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":523918720}
I20260806 20:15:47.628235 32578 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.628532 32565 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.617756 30676 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:47.629041 32565 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.630301 30676 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:47.631019 32238 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:47.631058 32565 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.631592 32238 raft_consensus.cc:686] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:47.629057 32568 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.632485 32238 consensus_queue.cc:237] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.604436 32620 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:47.633622 32638 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:47.555274 32610 consensus_queue.cc:1046] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.091s
I20260806 20:15:47.634127 32638 raft_consensus.cc:686] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:47.634424 32610 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:47.634927 32610 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:47.633579 30786 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1247) took 1522 ms (client timeout 1727 ms). Trace:
W20260806 20:15:47.635831 30786 rpcz_store.cc:269] 0806 20:15:46.110728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:47.514539 (+1403811us) service_pool.cc:227] Handling call
0806 20:15:47.633508 (+118969us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:47.628697 32585 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:47.637113 32610 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:47.537938 32612 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.677434 30711 raft_consensus.cc:1261] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:15:47.574018 32542 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.685406 32542 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.634999 32638 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.639281 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:47.639372 32569 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:47.636556 30676 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:47.641326 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 72968.5us
I20260806 20:15:47.643915 32271 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:47.644224 32613 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.645764 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 129) took 1541 ms. Trace:
I20260806 20:15:47.662078 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:47.662953 32614 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:47.663513 32078 raft_consensus.cc:3032] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:47.663729 32507 raft_consensus.cc:3032] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 20:15:47.663539 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 62451.1us
I20260806 20:15:47.627645 32443 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.609361 32370 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.666285 32426 consensus_queue.cc:1225] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer d4181d4d907f477f9feddc44ba3b9207 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:15:47.678918 32612 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.637676 32607 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:47.683272 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:47.687518 30602 catalog_manager.cc:5495] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 1 to 2, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 2 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
W20260806 20:15:47.698520 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 62058.8us
I20260806 20:15:47.721911 30676 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.700829 32542 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:47.731410 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.702772 30834 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.706111 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.713537 30792 raft_consensus.cc:1226] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 20:15:47.711480 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:47.628716 32578 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.670555 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1418) took 1143 ms. Trace:
I20260806 20:15:47.747252 30708 rpcz_store.cc:276] 0806 20:15:46.527276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.715376 (+188100us) service_pool.cc:227] Handling call
0806 20:15:47.670312 (+954936us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b54001b5b54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:47.670461 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1949299456}
I20260806 20:15:47.731328 32500 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:47.748682 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:47.752363 32426 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:47.753150 32426 consensus_queue.cc:1035] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:47.718051 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:47.719918 30709 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.003s sys 0.029s
I20260806 20:15:47.725875 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:47.754935 30709 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s user 0.003s sys 0.029s
W20260806 20:15:47.700765 30713 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.860s user 0.000s sys 0.030s
W20260806 20:15:47.755627 30713 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.914s user 0.000s sys 0.031s
I20260806 20:15:47.726280 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:47.756333 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 126) took 2142 ms. Trace:
I20260806 20:15:47.756704 30713 rpcz_store.cc:276] 0806 20:15:45.613849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.383684 (+769835us) service_pool.cc:227] Handling call
0806 20:15:46.383999 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:46.384210 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:15:46.841006 (+456796us) log.cc:842] Serialized 14 byte log entry
0806 20:15:47.756025 (+915019us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:47.756076 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:47.756115 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.756173 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.756281 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":580412,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"thread_start_us":1371103,"threads_started":2}
I20260806 20:15:47.757994 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.758797 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:47.759661 30713 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:47.760538 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:47.761281 32518 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:47.761782 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:47.761861 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:47.711342 32569 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.762476 30706 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:47.763016 30713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e"
dest_uuid: "42cabca48db7403b8523a3382a046043"
mode: GRACEFUL
new_leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
from {username='slave'} at 127.0.0.1:38516
I20260806 20:15:47.743727 32578 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:47.763547 30713 raft_consensus.cc:595] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Received request to transfer leadership to TS 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:47.763924 30713 raft_consensus.cc:603] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:47.764163 32569 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:47.766410 31414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48918 (request call id 311) took 2315 ms. Trace:
I20260806 20:15:47.764329 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:47.767771 32500 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:47.766837 31414 rpcz_store.cc:276] 0806 20:15:45.451346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:45.457300 (+ 5954us) service_pool.cc:227] Handling call
0806 20:15:47.144635 (+1687335us) spinlock_profiling.cc:243] Waited 8.32 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:47.159710 (+ 15075us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:47.399713 (+240003us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:47.399751 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:47.400250 (+ 499us) log.cc:842] Serialized 26 byte log entry
0806 20:15:47.614151 (+213901us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:47.625784 (+ 11633us) log.cc:842] Serialized 14 byte log entry
0806 20:15:47.678045 (+ 52261us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.678121 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:47.766080 (+ 87959us) raft_consensus.cc:1671] finished
0806 20:15:47.766183 (+ 103us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.766332 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44893,"prepare.queue_time_us":368,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":68661,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":191,"spinlock_wait_cycles":19127168,"thread_start_us":205120,"threads_started":1,"wal-append.queue_time_us":363831}
I20260806 20:15:47.770287 32518 consensus_queue.cc:1046] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:47.702754 30712 rpcz_store.cc:276] 0806 20:15:46.104866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.965396 (+860530us) service_pool.cc:227] Handling call
0806 20:15:46.965842 (+ 446us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:46.966395 (+ 553us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:46.966420 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:46.975708 (+ 9288us) log.cc:842] Serialized 26 byte log entry
0806 20:15:47.377500 (+401792us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:47.377574 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.377700 (+ 126us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:47.645518 (+267818us) raft_consensus.cc:1671] finished
0806 20:15:47.645539 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:47.645703 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":455309,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":346305,"threads_started":1,"wal-append.queue_time_us":550689}
I20260806 20:15:47.717291 32614 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.765232 30713 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:15:47.772109 30712 raft_consensus.cc:1261] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:15:47.772768 32614 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:47.676592 32608 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.735116 30711 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:47.727370 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.742897 32622 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:47.743010 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.742589 32611 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.743778 30676 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:47.726462 32274 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.969s user 0.014s sys 0.019s
I20260806 20:15:47.746001 30602 catalog_manager.cc:5495] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 1 to 4, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 4 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:47.756446 30870 raft_consensus.cc:2379] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:47.725840 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.770593 30601 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34006 (request call id 125) took 4715 ms. Trace:
I20260806 20:15:47.715456 32078 raft_consensus.cc:729] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:47.766665 31415 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEARNER]: Advancing to term 4
I20260806 20:15:47.703372 32639 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:47.714709 32507 raft_consensus.cc:729] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:47.773460 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.670625 32635 consensus_queue.cc:1035] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20260806 20:15:47.763688 30706 raft_consensus.cc:1261] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:15:47.770287 32443 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:47.749404 30708 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:47.742688 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.782907 30791 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
W20260806 20:15:47.787987 30757 leader_election.cc:334] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.681s (SENT)
I20260806 20:15:47.796706 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:47.797207 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 58044.3us
I20260806 20:15:47.798323 30756 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:47.827639 30756 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:47.802441 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 58791.4us
I20260806 20:15:47.801120 30867 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:47.804098 32614 consensus_queue.cc:1035] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:47.762005 32430 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.808481 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:47.806195 30602 catalog_manager.cc:5495] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:47.805810 32565 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:47.758153 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:47.800769 32274 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.076s user 0.015s sys 0.019s
I20260806 20:15:47.781579 30712 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:15:47.801395 32640 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:47.828894 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.805701 32510 raft_consensus.cc:3032] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:47.812652 30834 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.841027 32518 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:47.842384 30834 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.843168 32518 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:47.843902 30834 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.844650 32518 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:47.845819 30834 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:47.827747 31389 leader_election.cc:334] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.658s (SENT)
I20260806 20:15:47.846799 31389 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.799957 32608 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.825030 30869 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:15:47.704888 32545 consensus_queue.cc:260] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.807363 32611 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.805439 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:47.834431 32565 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:47.839884 30705 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:47.847157 30834 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.857908 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.858233 30834 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:47.859671 30676 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:47.847536 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:47.860610 30791 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:47.736326 32607 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.847945 30756 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:47.833658 32612 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:47.863787 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.830229 30711 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:47.739039 32542 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:47.816576 32618 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.855032 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.874241 30789 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:47.840611 32510 raft_consensus.cc:729] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:47.803812 32426 consensus_queue.cc:1046] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260806 20:15:47.864096 30602 catalog_manager.cc:5495] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 3 to 4, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 4 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:47.838122 32566 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:47.837604 32572 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.823532 32506 consensus_queue.cc:1035] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 20:15:47.801120 32621 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.906435 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:47.850317 32518 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:47.905419 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:47.912073 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 64201.8us
I20260806 20:15:47.918433 30867 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:15:47.919616 30871 raft_consensus.cc:1226] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:15:47.856878 32500 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:47.801775 30601 rpcz_store.cc:276] 0806 20:15:43.055207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.055397 (+ 190us) service_pool.cc:227] Handling call
0806 20:15:47.770395 (+4714998us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:15:44.107409 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:15:44.108262 (+ 853us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:15:44.108319 (+ 57us) write_op.cc:435] Acquired schema lock
0806 20:15:44.108350 (+ 31us) tablet.cc:660] Decoding operations
0806 20:15:44.334949 (+226599us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:15:44.351934 (+ 16985us) write_op.cc:641] Partition lock acquired for write op
0806 20:15:44.351977 (+ 43us) tablet.cc:683] Acquiring locks for 12 operations
0806 20:15:44.503011 (+151034us) tablet.cc:699] Row locks acquired
0806 20:15:44.503075 (+ 64us) write_op.cc:260] PREPARE: finished
0806 20:15:44.503375 (+ 300us) write_op.cc:270] Start()
0806 20:15:44.558547 (+ 55172us) write_op.cc:276] Timestamp: P: 1786047344503283 usec, L: 0
0806 20:15:44.564839 (+ 6292us) op_driver.cc:331] REPLICATION: starting
0806 20:15:44.576051 (+ 11212us) log.cc:842] Serialized 4786 byte log entry
0806 20:15:44.799325 (+223274us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f714e7355fe 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb
0806 20:15:45.159331 (+360006us) op_driver.cc:444] REPLICATION: finished
0806 20:15:45.419245 (+259914us) write_op.cc:301] APPLY: starting
0806 20:15:45.419359 (+ 114us) tablet.cc:1365] starting BulkCheckPresence
0806 20:15:45.422997 (+ 3638us) tablet.cc:1368] finished BulkCheckPresence
0806 20:15:45.423037 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:15:45.493514 (+ 70477us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:15:45.493851 (+ 337us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0806 20:15:45.493912 (+ 61us) write_op.cc:312] APPLY: finished
0806 20:15:45.494268 (+ 356us) log.cc:842] Serialized 89 byte log entry
0806 20:15:45.494431 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:45.494563 (+ 132us) write_op.cc:454] Released schema lock
0806 20:15:45.494631 (+ 68us) write_op.cc:341] FINISH: Updating metrics
0806 20:15:46.071966 (+577335us) spinlock_profiling.cc:243] Waited 5.19 ms on lock 0x7b5400002680. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f7154fb6361 00007f7155087254 00007f715508086b 00007f715507ffd1 00007f7155083dd1 00007f7155083d89 00007f7155083d19 00007f7155083ce1 00007f7155082fdd 00007f71579ab204 00007f71579ab039
0806 20:15:46.072072 (+ 106us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:46.072096 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":36152,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":200172,"apply.run_cpu_time_us":13317,"apply.run_wall_time_us":662207,"num_ops":12,"prepare.queue_time_us":28521,"prepare.run_cpu_time_us":20543,"prepare.run_wall_time_us":706573,"raft.queue_time_us":45172,"raft.run_cpu_time_us":8864,"raft.run_wall_time_us":231034,"replication_time_us":580434,"spinlock_wait_cycles":36093184,"thread_start_us":361888,"threads_started":4,"wal-append.queue_time_us":286188,"wal-append.run_cpu_time_us":6747,"wal-append.run_wall_time_us":1631875}]]}
I20260806 20:15:47.931003 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.932225 30870 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 20:15:47.934255 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.939324 30871 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:47.935002 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:47.940384 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.940187 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.941231 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.938251 30866 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: [3.5-4.6]
I20260806 20:15:47.856015 32608 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.863577 32565 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:47.863562 32443 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:47.852277 32640 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:47.952231 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 66191.5us
I20260806 20:15:47.946350 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.958017 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:47.982376 30835 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:47.982985 30869 raft_consensus.cc:2379] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:47.876547 32614 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:47.985368 32614 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:47.985870 32614 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:47.986270 32614 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:47.987118 32640 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:47.962128 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.959282 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:47.989612 32645 log.cc:924] Time spent T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260806 20:15:47.928610 32610 consensus_queue.cc:1046] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20260806 20:15:47.990399 32510 consensus_queue.cc:260] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.991340 32610 raft_consensus.cc:3032] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:47.991737 32610 raft_consensus.cc:729] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:47.992327 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.960623 30790 raft_consensus.cc:2379] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:47.992627 32610 consensus_queue.cc:260] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.876317 32612 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:47.800495 32622 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:47.910216 32609 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:47.978204 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 50379.7us
I20260806 20:15:47.979763 30870 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 20:15:47.855831 32345 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.077s
I20260806 20:15:47.988929 30864 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 20:15:47.993569 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:47.924782 30602 catalog_manager.cc:5495] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 2 to 3, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:47.994676 32607 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:47.994812 32238 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:47.995980 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.024531 30792 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:47.997220 30789 raft_consensus.cc:1226] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:47.989670 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:15:47.982340 30865 raft_consensus.cc:2362] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5. Re-sending same reply.
I20260806 20:15:47.997138 30868 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:15:48.001679 32443 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:47.944108 32613 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:47.911620 32618 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:48.018003 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 97632.2us
I20260806 20:15:48.028702 30757 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:48.014622 32612 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:48.029954 30602 catalog_manager.cc:5495] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 2 to 3, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
W20260806 20:15:48.021696 32627 log.cc:924] Time spent T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260806 20:15:47.926244 32616 consensus_queue.cc:1035] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.096s
I20260806 20:15:48.033812 32631 consensus_queue.cc:1046] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260806 20:15:48.022215 32622 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.910142 32444 consensus_queue.cc:1035] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.101s
I20260806 20:15:47.977440 32500 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:48.035403 32500 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.036966 32611 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:47.931092 32642 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:48.031646 30602 catalog_manager.cc:5495] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 2 to 3, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:48.032939 30790 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:48.037817 32642 raft_consensus.cc:686] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:48.038631 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.034888 31415 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEARNER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:15:48.005473 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:48.048115 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.040328 30791 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:47.932768 32580 consensus_queue.cc:1035] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
I20260806 20:15:48.005271 32566 raft_consensus.cc:686] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:48.034837 32631 raft_consensus.cc:3032] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:15:48.065542 30757 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:48.066470 32013 raft_consensus.cc:2741] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260806 20:15:48.067130 32013 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:48.068985 30757 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:48.070631 32640 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.037753 32500 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.078910 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:48.079051 30676 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:48.075301 32574 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:48.081727 30676 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:48.081768 32611 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:48.039047 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 132) took 1362 ms. Trace:
W20260806 20:15:48.086180 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 275590us
I20260806 20:15:48.049515 30784 raft_consensus.cc:1226] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:48.084964 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.088732 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:47.928493 32572 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.091490 30865 raft_consensus.cc:2396] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:48.054482 30786 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.045727 32610 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.039860 32443 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.107148 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.089681 32622 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:48.113396 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.082351 32611 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:48.039722 32238 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:48.107918 32274 consensus_queue.cc:1035] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260806 20:15:48.147079 32238 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:48.147626 32630 consensus_queue.cc:1035] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
I20260806 20:15:48.121021 30871 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:15:48.123158 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.151535 30869 raft_consensus.cc:2396] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:48.077374 32631 raft_consensus.cc:729] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:48.124652 30792 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:48.093874 32430 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.152895 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1412) took 1629 ms. Trace:
W20260806 20:15:48.113221 32636 log.cc:924] Time spent T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20260806 20:15:48.153376 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:48.152876 32631 consensus_queue.cc:260] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.155279 32631 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:15:48.155985 32650 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260806 20:15:48.153312 30709 rpcz_store.cc:276] 0806 20:15:46.523912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:47.066891 (+542979us) service_pool.cc:227] Handling call
0806 20:15:47.126820 (+ 59929us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:47.127059 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:47.127085 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:47.145033 (+ 17948us) log.cc:842] Serialized 26 byte log entry
0806 20:15:47.755346 (+610313us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:47.755383 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.755448 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:48.152706 (+397258us) raft_consensus.cc:1671] finished
0806 20:15:48.152726 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:48.152842 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":592853,"threads_started":1,"wal-append.queue_time_us":691252}
I20260806 20:15:48.158667 32443 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:48.123158 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:48.086414 30703 rpcz_store.cc:276] 0806 20:15:46.667925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:47.416740 (+748815us) service_pool.cc:227] Handling call
0806 20:15:47.417196 (+ 456us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:47.425594 (+ 8398us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:47.425626 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:47.426150 (+ 524us) log.cc:842] Serialized 26 byte log entry
0806 20:15:47.722181 (+296031us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:47.730805 (+ 8624us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:47.742192 (+ 11387us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:48.014691 (+272499us) raft_consensus.cc:1671] finished
0806 20:15:48.014722 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:48.030571 (+ 15849us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7658,"raft.queue_time_us":7785,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":285189,"threads_started":1,"wal-append.queue_time_us":397008}
I20260806 20:15:48.125051 30756 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:48.166572 30784 raft_consensus.cc:1261] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:48.132150 30866 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:48.135955 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:15:48.092408 32013 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260806 20:15:48.170276 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.079234 32607 consensus_queue.cc:1035] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20260806 20:15:48.171625 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.109037 32542 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.172003 32608 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.170301 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:15:48.174111 32566 consensus_queue.cc:237] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.144950 32618 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.177659 32642 consensus_queue.cc:237] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.082059 32574 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:48.194139 30757 leader_election.cc:334] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.549s (SENT)
I20260806 20:15:48.194717 30757 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:48.197336 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 215046us
I20260806 20:15:48.198177 30835 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:48.199385 30835 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.204737 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.130970 32617 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:48.180887 32611 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:48.158900 32572 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.216593 30835 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.169487 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:48.220504 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 95542.1us
I20260806 20:15:48.132695 32622 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:48.189520 32638 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:48.228673 30756 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:48.229365 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 401632us
I20260806 20:15:48.221904 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:48.234817 30835 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.236451 30835 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:48.183179 32518 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.242345 30712 raft_consensus.cc:2379] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:48.242894 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:48.243585 30713 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:48.133020 32345 consensus_queue.cc:1035] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20260806 20:15:48.251778 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.242101 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:48.184526 32568 consensus_queue.cc:1035] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20260806 20:15:48.218605 32530 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:48.215411 32506 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:48.227348 32607 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:48.296249 30785 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:48.296427 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:48.241649 32622 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:48.224292 32621 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.249917 32580 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.267288 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:48.315773 30713 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:48.258472 31389 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.269335 30756 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:48.227164 32013 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.257736 30867 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:48.260286 30784 raft_consensus.cc:1261] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 20:15:48.303123 30788 raft_consensus.cc:1261] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260806 20:15:48.303656 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.241232 32652 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.254812 32578 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:48.263334 30710 raft_consensus.cc:1226] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:48.342046 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:48.268352 32530 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:48.347877 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 78663.9us
I20260806 20:15:48.341799 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:48.353243 30788 raft_consensus.cc:2379] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:48.353358 30756 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:48.349795 30866 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.231377 32617 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:48.310863 32518 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:48.320438 32569 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.374070 30785 raft_consensus.cc:1261] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:48.250293 32638 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.377007 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:48.377635 31388 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:48.358026 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:48.389954 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 131551us
I20260806 20:15:48.304405 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1279) took 1592 ms. Trace:
I20260806 20:15:48.338486 30864 raft_consensus.cc:2379] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:48.348403 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:48.353715 32468 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:48.431252 30784 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:48.374207 31417 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:15:48.371869 30703 raft_consensus.cc:1261] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:48.371496 30705 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:48.450482 32616 consensus_queue.cc:1035] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:48.452005 32616 consensus_queue.cc:1046] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:48.452731 32616 consensus_queue.cc:1225] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer d4181d4d907f477f9feddc44ba3b9207 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:15:48.453790 30756 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:48.371169 32574 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:48.347048 32426 consensus_queue.cc:1035] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260806 20:15:48.394127 32607 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:48.354164 32613 consensus_queue.cc:1035] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20260806 20:15:48.433552 32468 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:48.355664 32614 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.394659 32518 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.366389 32649 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:48.434635 30788 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:48.449695 32443 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:48.455637 30709 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:48.355427 32530 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.416307 32618 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:48.458762 30787 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260806 20:15:48.431303 32646 log.cc:924] Time spent T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260806 20:15:48.372912 32664 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:48.426689 32542 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:48.454141 30785 raft_consensus.cc:1261] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:48.453678 32534 consensus_queue.cc:1035] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260806 20:15:48.480963 32511 consensus_queue.cc:1035] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260806 20:15:48.497762 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.413908 32652 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260806 20:15:48.463289 32607 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:48.498732 32607 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:48.464390 32513 consensus_meta.cc:287] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.910s user 0.012s sys 0.001s
I20260806 20:15:48.499012 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.499301 32513 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:48.500264 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.501006 30864 raft_consensus.cc:2379] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:48.473363 30756 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:48.501543 30868 raft_consensus.cc:1226] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:48.493531 32542 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:48.502116 32542 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.493065 32664 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.503048 32574 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:48.494194 32614 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:48.462380 32468 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.481863 32649 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:48.417688 30863 rpcz_store.cc:276] 0806 20:15:46.704993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.705999 (+ 1006us) service_pool.cc:227] Handling call
0806 20:15:46.706433 (+ 434us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:47.431650 (+725217us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:47.431681 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:47.432275 (+ 594us) log.cc:842] Serialized 52 byte log entry
0806 20:15:47.806361 (+374086us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:15:48.125923 (+319562us) log.cc:842] Serialized 14 byte log entry
0806 20:15:48.193482 (+ 67559us) log.cc:842] Serialized 14 byte log entry
0806 20:15:48.233841 (+ 40359us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:48.255975 (+ 22134us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:48.258061 (+ 2086us) raft_consensus.cc:1671] finished
0806 20:15:48.262518 (+ 4457us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:48.297543 (+ 35025us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17481,"prepare.queue_time_us":528289,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":83,"raft.queue_time_us":219765,"raft.run_cpu_time_us":518,"raft.run_wall_time_us":61518,"thread_start_us":1337985,"threads_started":3,"wal-append.queue_time_us":473726}
I20260806 20:15:48.495227 32518 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:48.504696 30834 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.507295 30757 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:48.508524 30866 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:48.505625 30756 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:48.504961 32574 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:48.523737 30756 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:48.445216 32388 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.039s
I20260806 20:15:48.482259 32608 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:48.522099 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:48.522305 30676 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:48.481904 30786 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:48.549695 30790 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:48.550486 30676 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.490876 32617 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:48.564972 32612 consensus_meta.cc:287] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.517s user 0.007s sys 0.005s
I20260806 20:15:48.565423 32612 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.490665 32631 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:48.566448 32545 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:48.566890 32676 log.cc:924] Time spent T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260806 20:15:48.566648 32631 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:48.484088 32444 consensus_queue.cc:1046] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
W20260806 20:15:48.567366 32660 log.cc:924] Time spent T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.213s user 0.000s sys 0.000s
I20260806 20:15:48.567931 32444 consensus_queue.cc:1046] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 20:15:48.569566 32631 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:48.570096 32444 consensus_queue.cc:1046] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:48.570858 32652 raft_consensus.cc:3032] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:48.571204 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:48.571923 30861 raft_consensus.cc:2362] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5. Re-sending same reply.
I20260806 20:15:48.572436 32638 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.572829 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.572949 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.536669 32640 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:48.573781 32640 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:48.574162 32640 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.574467 32510 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:48.484036 32345 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
W20260806 20:15:48.575794 32641 log.cc:924] Time spent T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260806 20:15:48.567873 30791 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:48.577883 30864 raft_consensus.cc:2379] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:48.578122 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.571208 32652 raft_consensus.cc:729] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:48.579341 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:48.579895 30791 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.578774 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.561336 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.563809 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.565773 32671 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 20:15:48.565907 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 44855.4us
I20260806 20:15:48.570197 31415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 326) took 1339 ms. Trace:
I20260806 20:15:48.581230 31415 rpcz_store.cc:276] 0806 20:15:47.232580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:47.238288 (+ 5708us) service_pool.cc:227] Handling call
0806 20:15:47.766537 (+528249us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:48.049081 (+282544us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:48.569985 (+520904us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b5400602550. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:48.570135 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1111442176}
I20260806 20:15:48.581751 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.582569 30786 raft_consensus.cc:2379] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:48.584383 30676 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.504485 32664 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.579767 32652 consensus_queue.cc:260] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.531141 32614 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:48.501360 32513 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.586079 30676 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:48.581151 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:48.530282 32574 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:48.547714 32608 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:48.601305 32608 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:48.530483 32534 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.536108 32585 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:48.503937 32621 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260806 20:15:48.576380 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.481668 32565 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
W20260806 20:15:48.597743 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 90526us
I20260806 20:15:48.576489 30705 raft_consensus.cc:1261] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260806 20:15:48.598803 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:48.600142 30676 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:48.588673 32500 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.594004 30864 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:48.600642 32672 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:48.600840 32238 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:48.523881 32613 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260806 20:15:48.588091 32671 raft_consensus.cc:686] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:48.609519 32668 consensus_queue.cc:1046] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260806 20:15:48.609766 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:48.610800 32680 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.023s
I20260806 20:15:48.611573 32607 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.618324 30757 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:48.620096 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:48.591615 32616 consensus_queue.cc:1035] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260806 20:15:48.613139 32585 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:48.618959 30705 pending_rounds.cc:76] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 1
W20260806 20:15:48.619493 32610 consensus_meta.cc:287] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.514s user 0.007s sys 0.003s
I20260806 20:15:48.654796 32664 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 20:15:48.659004 31389 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.527s (SENT) [suppressed 26 similar messages]
I20260806 20:15:48.680858 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.672361 30863 raft_consensus.cc:2379] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:48.693158 32672 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:48.697157 32672 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.699551 32672 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.670542 32642 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:48.655368 32238 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:48.660126 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:48.614704 32652 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.660024 30832 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:48.710031 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 50108.1us
W20260806 20:15:48.665680 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 47555.5us
W20260806 20:15:48.671689 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 51679.2us
I20260806 20:15:48.622237 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.671763 30791 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:48.711159 30676 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.711472 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.711714 30835 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:48.666518 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.678440 30705 pending_rounds.cc:97] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:15:48.676317 32585 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:48.712157 30757 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:48.714246 30757 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:48.653065 30756 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:48.709247 30866 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:48.718771 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.718573 30676 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.717077 30865 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:48.724937 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:48.740411 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 116883us
W20260806 20:15:48.762475 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 50842.1us
I20260806 20:15:48.763183 30757 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:48.764824 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 49737.4us
I20260806 20:15:48.765770 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.766731 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.705926 32610 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:48.769913 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 110950us
I20260806 20:15:48.674276 32345 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20260806 20:15:48.713750 32238 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:48.774559 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.776943 30865 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:48.777868 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:48.707443 32664 raft_consensus.cc:686] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:48.714366 30705 raft_consensus.cc:2866] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:48.724054 32672 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:48.765772 32013 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.783878 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:48.762014 32511 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:48.766997 32274 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:48.772209 32667 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:48.774892 32612 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.788211 30861 raft_consensus.cc:2396] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:48.807392 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:48.800299 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 80050.4us
I20260806 20:15:48.765475 32510 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:48.779245 32621 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:48.815059 31389 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:48.818588 32693 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:48.781526 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1533) took 1058 ms. Trace:
I20260806 20:15:48.813719 32618 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.009s
I20260806 20:15:48.823038 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:48.822198 30866 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:48.747560 32542 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:48.856078 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:48.808079 32672 raft_consensus.cc:686] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:48.776072 32610 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:48.858597 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:48.803402 32649 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.799865 32608 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.854861 32565 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.845953 32274 raft_consensus.cc:686] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:48.878780 32694 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:48.879290 32663 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:48.879882 32542 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:48.880060 32690 log.cc:924] Time spent T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260806 20:15:48.880267 32691 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:48.879420 32694 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.847453 32667 raft_consensus.cc:686] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:48.884977 32628 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:48.860466 32612 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:48.883213 32664 consensus_queue.cc:237] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:48.886132 32688 log.cc:924] Time spent T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20260806 20:15:48.881052 32673 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:48.886695 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:48.886129 32274 consensus_queue.cc:237] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.888872 32608 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.889379 30791 raft_consensus.cc:1226] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:48.889283 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.907855 30785 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:15:48.893493 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 78498.9us
I20260806 20:15:48.885290 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:48.908725 30710 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:48.825132 30783 rpcz_store.cc:276] 0806 20:15:47.721864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:47.726683 (+ 4819us) service_pool.cc:227] Handling call
0806 20:15:48.614312 (+887629us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:48.779733 (+165421us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86535552}
I20260806 20:15:48.879807 32663 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:48.855808 32322 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260806 20:15:48.908957 31389 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:48.911146 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:48.913434 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:49.121587 30709 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:48.913795 32672 consensus_queue.cc:237] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:48.916812 32622 consensus_meta.cc:287] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.576s user 0.006s sys 0.002s
I20260806 20:15:48.918396 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:48.921998 32667 consensus_queue.cc:237] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:48.819270 32693 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:48.897143 32607 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:48.928758 30835 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:48.958953 30757 leader_election.cc:334] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.543s (SENT)
W20260806 20:15:49.187152 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 228264us
I20260806 20:15:49.130857 30676 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:49.120242 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 109) took 1203 ms. Trace:
W20260806 20:15:49.147787 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 238893us
I20260806 20:15:48.779485 32671 consensus_queue.cc:237] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.153427 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:49.180876 31422 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:49.188879 30867 rpcz_store.cc:276] 0806 20:15:47.916780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:47.940996 (+ 24216us) service_pool.cc:227] Handling call
0806 20:15:49.120029 (+1179033us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b540007d854. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:49.120167 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":726333312}
I20260806 20:15:49.191088 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:49.190934 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 60215.3us
I20260806 20:15:49.195247 30867 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:49.196422 30785 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:49.117805 32663 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:49.149108 32578 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:49.186369 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 257708us
I20260806 20:15:49.218456 31389 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:49.177130 32618 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:49.225898 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:49.233045 30867 raft_consensus.cc:2396] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:49.223140 30712 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:49.235906 30785 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:49.208663 30709 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:49.238322 30756 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:49.239269 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:49.239948 30709 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:49.242654 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:49.210283 32639 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:49.253837 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:49.147938 30705 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:49.209834 32616 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:49.272061 30710 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:49.274461 31388 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:49.307346 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:49.309621 30792 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:49.182484 32622 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:49.318041 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 43646.9us
I20260806 20:15:49.319785 31388 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:49.213063 32617 consensus_queue.cc:1046] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260806 20:15:49.242590 32511 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:49.252708 32642 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:49.331277 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1291) took 1247 ms. Trace:
I20260806 20:15:49.331714 30868 rpcz_store.cc:276] 0806 20:15:48.083893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.084026 (+ 133us) service_pool.cc:227] Handling call
0806 20:15:48.084383 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:48.501989 (+417606us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.331012 (+829023us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54002fbcd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:49.331218 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":921858304}
I20260806 20:15:49.334090 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:49.334892 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1165) took 1204 ms (client timeout 1535 ms). Trace:
W20260806 20:15:49.335245 30869 rpcz_store.cc:269] 0806 20:15:48.130004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.152057 (+ 22053us) service_pool.cc:227] Handling call
0806 20:15:49.333921 (+1181864us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:49.334829 (+ 908us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46571264}
I20260806 20:15:49.269054 32616 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:49.343312 30757 leader_election.cc:334] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.535s (SENT)
I20260806 20:15:49.343461 32647 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:49.343693 32271 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:49.344072 32647 raft_consensus.cc:686] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:49.344228 32271 raft_consensus.cc:686] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:49.301008 32612 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.346730 30713 raft_consensus.cc:1226] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
W20260806 20:15:49.328074 32585 consensus_meta.cc:287] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.567s user 0.007s sys 0.004s
I20260806 20:15:49.246929 32578 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:49.347537 32610 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:49.270327 32621 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260806 20:15:49.347759 32616 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:49.349494 32642 raft_consensus.cc:686] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:49.362728 30709 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:49.345273 32271 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.356346 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:49.408788 30709 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:49.357771 32694 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:49.412662 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 194296us
I20260806 20:15:49.342857 32511 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.258047 32628 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.141s
W20260806 20:15:49.408844 32530 consensus_meta.cc:287] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.914s user 0.011s sys 0.000s
I20260806 20:15:49.421420 30792 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:49.362500 32585 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.434640 32370 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.358770 32612 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.435420 32370 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:49.432626 32699 log.cc:924] Time spent T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 20:15:49.431193 32078 consensus_queue.cc:260] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:49.426901 32696 log.cc:924] Time spent T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20260806 20:15:49.345075 32647 consensus_queue.cc:237] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:49.438048 31415 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.010s sys 0.035s
I20260806 20:15:49.420122 30705 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:49.467782 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:49.457047 31415 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s user 0.011s sys 0.035s
I20260806 20:15:49.461676 30786 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:15:49.427414 32530 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.460790 32585 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.481407 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:49.482306 30792 raft_consensus.cc:2379] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:49.479149 30676 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:49.481649 30835 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:49.485857 30792 raft_consensus.cc:1226] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 20:15:49.445415 32370 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.477532 32621 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.476022 32693 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:49.486625 32468 consensus_meta.cc:287] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.963s user 0.009s sys 0.002s
I20260806 20:15:49.501150 32612 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.501303 32468 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.502220 32693 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:49.502753 32693 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.503507 32468 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.504608 32693 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.505098 32693 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:49.489298 32618 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:49.452821 32507 consensus_queue.cc:260] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.506084 32507 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:49.506516 30784 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:49.506376 32693 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.508489 32693 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.500110 30705 raft_consensus.cc:1261] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:49.509349 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1446) took 1021 ms. Trace:
I20260806 20:15:49.464538 32642 consensus_queue.cc:237] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.509720 30705 rpcz_store.cc:276] 0806 20:15:48.487730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.472526 (+984796us) service_pool.cc:227] Handling call
0806 20:15:49.495507 (+ 22981us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:49.509067 (+ 13560us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.509290 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:49.500759 32370 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.499720 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1438) took 1279 ms. Trace:
I20260806 20:15:49.512440 32370 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:49.513512 32691 consensus_meta.cc:287] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.633s user 0.004s sys 0.005s
I20260806 20:15:49.512313 30708 rpcz_store.cc:276] 0806 20:15:48.220433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.224521 (+ 4088us) service_pool.cc:227] Handling call
0806 20:15:48.784810 (+560289us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:48.785055 (+ 245us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:48.823926 (+ 38871us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:48.827042 (+ 3116us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:48.878865 (+ 51823us) log.cc:842] Serialized 26 byte log entry
0806 20:15:49.145656 (+266791us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:49.155076 (+ 9420us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.186313 (+ 31237us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:49.494567 (+308254us) raft_consensus.cc:1671] finished
0806 20:15:49.496531 (+ 1964us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:49.499634 (+ 3103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3468,"raft.queue_time_us":32796,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":122,"spinlock_wait_cycles":29744384,"thread_start_us":45161,"threads_started":1,"wal-append.queue_time_us":432087}
W20260806 20:15:49.514892 32545 consensus_meta.cc:287] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.927s user 0.006s sys 0.005s
W20260806 20:15:49.515132 32652 consensus_meta.cc:287] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.791s user 0.009s sys 0.002s
I20260806 20:15:49.531118 32652 raft_consensus.cc:2726] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:15:49.532209 32238 consensus_meta.cc:287] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.744s user 0.008s sys 0.001s
I20260806 20:15:49.532876 32530 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:49.513921 32691 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:15:49.534291 32611 consensus_meta.cc:287] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.293s user 0.008s sys 0.004s
I20260806 20:15:49.534675 32611 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.536139 32663 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.536679 32611 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.543403 31423 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:49.544122 32649 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.544585 32649 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.546705 32578 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:49.546849 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1539) took 1817 ms. Trace:
I20260806 20:15:49.547271 30787 rpcz_store.cc:276] 0806 20:15:47.729711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.150352 (+420641us) service_pool.cc:227] Handling call
0806 20:15:48.150762 (+ 410us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:48.484524 (+333762us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.546556 (+1062032us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54002dfad4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:49.546786 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707749120}
I20260806 20:15:49.547801 30870 raft_consensus.cc:2396] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:49.548272 32510 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.548746 32510 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.549237 32663 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.551308 32616 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.551821 30707 raft_consensus.cc:1226] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:49.487553 32622 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:49.552479 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1432) took 1333 ms. Trace:
I20260806 20:15:49.554032 30707 rpcz_store.cc:276] 0806 20:15:48.218756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.232305 (+ 13549us) service_pool.cc:227] Handling call
0806 20:15:49.551408 (+1319103us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:49.551656 (+ 248us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:49.552225 (+ 569us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.552421 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5894016}
W20260806 20:15:49.536600 32534 consensus_meta.cc:287] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.923s user 0.010s sys 0.001s
I20260806 20:15:49.532666 32238 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.508836 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1558) took 1127 ms. Trace:
W20260806 20:15:49.555457 32574 consensus_meta.cc:287] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.941s user 0.011s sys 0.000s
I20260806 20:15:49.498262 32621 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:49.556526 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 236829us
I20260806 20:15:49.573064 30706 raft_consensus.cc:2379] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:49.610155 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1146) took 1489 ms. Trace:
I20260806 20:15:49.596808 32511 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:49.609048 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:49.638115 32616 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:49.572863 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 326882us
I20260806 20:15:49.640240 30756 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:49.579692 30676 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:49.557356 32675 log.cc:924] Time spent T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.938s user 0.005s sys 0.011s
I20260806 20:15:49.641758 30756 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:49.557013 32534 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:49.680497 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:49.636909 30865 rpcz_store.cc:276] 0806 20:15:48.121210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.789294 (+668084us) service_pool.cc:227] Handling call
0806 20:15:48.789664 (+ 370us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:48.790034 (+ 370us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:48.790062 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:48.790509 (+ 447us) log.cc:842] Serialized 26 byte log entry
0806 20:15:49.310793 (+520284us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:49.340654 (+ 29861us) log.cc:842] Serialized 14 byte log entry
0806 20:15:49.609802 (+269148us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.609887 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:49.609937 (+ 50us) raft_consensus.cc:1671] finished
0806 20:15:49.609950 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:49.610090 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4733,"prepare.queue_time_us":30949,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":6535,"raft.queue_time_us":174988,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":44301,"thread_start_us":729488,"threads_started":2,"wal-append.queue_time_us":589709}
W20260806 20:15:49.682600 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 103005us
W20260806 20:15:49.599823 32673 consensus_meta.cc:287] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.689s user 0.005s sys 0.004s
I20260806 20:15:49.699461 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:49.702718 32717 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.682636 32707 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.703402 30865 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:49.707449 32568 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:49.707875 32568 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:49.708196 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1442) took 1239 ms. Trace:
I20260806 20:15:49.706933 32717 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.710075 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:49.709880 32707 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.710702 32717 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.704777 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:49.712031 32707 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.711248 32717 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.712658 32707 consensus_queue.cc:1046] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:49.714499 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:49.714813 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:49.712234 30710 rpcz_store.cc:276] 0806 20:15:48.468748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.331192 (+862444us) service_pool.cc:227] Handling call
0806 20:15:49.332012 (+ 820us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:49.332751 (+ 739us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:49.332818 (+ 67us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:49.333782 (+ 964us) log.cc:842] Serialized 26 byte log entry
0806 20:15:49.555777 (+221995us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:49.555813 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.565319 (+ 9506us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:49.707984 (+142665us) raft_consensus.cc:1671] finished
0806 20:15:49.708004 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:49.708133 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":102640,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"thread_start_us":221343,"threads_started":1,"wal-append.queue_time_us":345476}
I20260806 20:15:49.716516 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:49.717077 30710 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:49.718771 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:49.708403 32568 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:49.720697 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:49.693041 32693 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:49.569700 32574 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.719185 32717 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.640094 32507 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.710707 30790 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:49.741518 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:49.588197 32580 consensus_queue.cc:1046] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.104s
I20260806 20:15:49.721356 30865 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:49.692945 32511 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 20:15:49.703548 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 61879.5us
I20260806 20:15:49.745726 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:49.679065 32691 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:49.748600 30865 raft_consensus.cc:2396] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:49.680855 32611 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:49.679360 32500 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:49.763429 30788 raft_consensus.cc:1226] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:15:49.763442 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:49.764209 30790 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:49.723199 32710 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:49.783694 30835 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:49.679169 32708 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.780202 30865 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:15:49.788610 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1152) took 1665 ms. Trace:
I20260806 20:15:49.723834 32673 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:49.748265 32511 raft_consensus.cc:686] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:49.748447 32664 raft_consensus.cc:3032] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:49.788968 30865 rpcz_store.cc:276] 0806 20:15:48.122668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.779502 (+1656834us) service_pool.cc:227] Handling call
0806 20:15:49.779847 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:49.788328 (+ 8481us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.788552 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:49.778658 32580 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:49.743685 32693 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.745538 32507 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.825644 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1538) took 2096 ms. Trace:
I20260806 20:15:49.557538 30784 rpcz_store.cc:276] 0806 20:15:48.381387 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.432206 (+ 50819us) service_pool.cc:227] Handling call
0806 20:15:48.432510 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:49.506944 (+1074434us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.508591 (+ 1647us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400346054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:49.508775 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2469577088}
W20260806 20:15:49.832588 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 132813us
W20260806 20:15:49.849880 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 217194us
W20260806 20:15:49.850554 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 131774us
I20260806 20:15:49.817432 32238 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.766556 32611 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.849200 30676 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:49.822499 32370 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:49.790650 32718 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:49.862147 30676 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:49.848610 30789 rpcz_store.cc:276] 0806 20:15:47.729423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.052684 (+323261us) service_pool.cc:227] Handling call
0806 20:15:48.083599 (+ 30915us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:48.087282 (+ 3683us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:48.087310 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:48.105237 (+ 17927us) log.cc:842] Serialized 26 byte log entry
0806 20:15:48.257350 (+152113us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:48.257387 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:48.257471 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:49.510816 (+1253345us) spinlock_profiling.cc:243] Waited 63 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:15:49.821090 (+310274us) raft_consensus.cc:1671] finished
0806 20:15:49.821123 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:49.825565 (+ 4442us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55103,"raft.queue_time_us":24773,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":13520,"spinlock_wait_cycles":144851840,"thread_start_us":115612,"threads_started":1,"wal-append.queue_time_us":415657}
I20260806 20:15:49.820664 32664 raft_consensus.cc:729] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:49.789917 32710 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.863278 32647 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260806 20:15:49.784194 32691 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.862799 30710 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
W20260806 20:15:49.855430 32640 consensus_meta.cc:287] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.259s user 0.011s sys 0.002s
I20260806 20:15:49.849802 32673 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:49.889753 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:49.874631 32667 consensus_queue.cc:1046] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.020s
I20260806 20:15:49.829421 32580 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:49.887667 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:49.817910 32708 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.892997 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:50.792445 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:49.895229 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:50.286007 32542 raft_consensus.cc:3032] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:50.794308 32542 raft_consensus.cc:729] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
W20260806 20:15:50.134478 32720 log.cc:924] Time spent T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
W20260806 20:15:50.629554 30835 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.009s sys 0.032s
W20260806 20:15:50.795697 30835 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.009s sys 0.039s
I20260806 20:15:50.224144 32507 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.796705 32708 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:50.054699 32705 log.cc:924] Time spent T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.325s user 0.001s sys 0.001s
I20260806 20:15:50.293869 32616 raft_consensus.cc:3032] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:50.798610 32616 raft_consensus.cc:729] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:50.799161 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1144) took 2678 ms. Trace:
I20260806 20:15:49.892356 32640 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.248713 32611 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.237677 32712 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:15:50.252035 31423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 340) took 1875 ms (client timeout 2059 ms). Trace:
I20260806 20:15:50.799635 30864 rpcz_store.cc:276] 0806 20:15:48.120731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.778863 (+658132us) service_pool.cc:227] Handling call
0806 20:15:48.797265 (+ 18402us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:48.820136 (+ 22871us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:48.823697 (+ 3561us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:48.899412 (+ 75715us) log.cc:842] Serialized 26 byte log entry
0806 20:15:49.488940 (+589528us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:49.560805 (+ 71865us) log.cc:842] Serialized 14 byte log entry
0806 20:15:49.574235 (+ 13430us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.585934 (+ 11699us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:50.798902 (+1212968us) raft_consensus.cc:1671] finished
0806 20:15:50.798929 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:50.799088 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1900,"prepare.queue_time_us":32289,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":41,"raft.queue_time_us":6426,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":12293,"thread_start_us":269683,"threads_started":1,"wal-append.queue_time_us":454961}
W20260806 20:15:50.801430 31423 rpcz_store.cc:269] 0806 20:15:48.376569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.376732 (+ 163us) service_pool.cc:227] Handling call
0806 20:15:50.251864 (+1875132us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540040ef54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:50.251978 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2643389696}
W20260806 20:15:50.139307 32725 log.cc:924] Time spent T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.231s user 0.000s sys 0.001s
I20260806 20:15:50.802595 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:49.991891 32574 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:49.998540 32238 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:50.590939 32710 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.803893 32752 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:50.804064 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:50.295236 32677 log.cc:924] Time spent T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 1.628s user 0.000s sys 0.014s
I20260806 20:15:50.249672 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1565) took 1740 ms. Trace:
W20260806 20:15:50.805768 32565 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.003s sys 0.012s
I20260806 20:15:49.911284 30871 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:50.431952 31421 raft_consensus.cc:1226] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:50.073024 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:50.813938 30865 raft_consensus.cc:1261] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:15:50.114444 30676 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:50.138798 32724 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260806 20:15:50.288786 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 143) took 1870 ms. Trace:
W20260806 20:15:50.242334 32726 log.cc:924] Time spent T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.214s user 0.000s sys 0.001s
W20260806 20:15:50.138131 32730 log.cc:924] Time spent T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260806 20:15:50.805299 30791 rpcz_store.cc:276] 0806 20:15:48.509563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.775080 (+265517us) service_pool.cc:227] Handling call
0806 20:15:48.775376 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:48.909217 (+133841us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:50.249418 (+1340201us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54002df5d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:50.249611 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":261710720}
I20260806 20:15:50.804785 30864 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:50.249317 32691 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:50.072782 32716 log.cc:924] Time spent T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
W20260806 20:15:50.849819 30754 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.008s sys 0.007s
I20260806 20:15:50.801225 32712 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:49.890435 32500 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:50.823988 32752 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.851182 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1445) took 2373 ms. Trace:
I20260806 20:15:49.970366 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1148) took 1426 ms. Trace:
I20260806 20:15:50.852555 32752 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.851926 30708 rpcz_store.cc:276] 0806 20:15:48.544300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.514226 (+969926us) service_pool.cc:227] Handling call
0806 20:15:49.514526 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:49.514719 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:49.791014 (+276295us) log.cc:842] Serialized 14 byte log entry
0806 20:15:49.969890 (+178876us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:49.969947 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:49.969990 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.970064 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:49.970287 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":414365,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":292513,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":449244,"threads_started":2}
I20260806 20:15:50.294390 32698 raft_consensus.cc:3032] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:50.823222 30711 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
W20260806 20:15:50.253608 32568 consensus_meta.cc:287] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.533s user 0.010s sys 0.000s
W20260806 20:15:50.250815 32681 log.cc:924] Time spent T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 1.647s user 0.007s sys 0.009s
I20260806 20:15:50.119571 32530 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:50.247179 32731 log.cc:924] Time spent T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.319s user 0.000s sys 0.000s
W20260806 20:15:50.828341 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 713967us
W20260806 20:15:50.830117 30871 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1143) took 2693 ms (client timeout 1543 ms). Trace:
I20260806 20:15:50.844683 32574 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:50.813476 32565 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.740s user 0.004s sys 0.013s
I20260806 20:15:50.224529 32693 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.768249 32510 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:50.134104 32713 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:49.928920 32621 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:50.531906 32668 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:50.892503 32668 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.039997 32717 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:50.894882 32668 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.042099 32664 consensus_queue.cc:260] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.895829 32430 consensus_queue.cc:1046] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:50.896469 32712 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.896624 32640 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:50.897228 32640 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:50.897768 32640 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.899372 32758 raft_consensus.cc:1050] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:50.899744 32640 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:50.899885 32758 raft_consensus.cc:1054] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Unable to promote non-voter 58f2f6256ee14cb38a1dbcbf5bb590a0: Illegal state: Leader has not yet committed an operation in its own term
I20260806 20:15:50.900303 32640 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:50.799724 32616 consensus_queue.cc:260] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.901834 32616 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:50.903194 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:50.903815 30864 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:50.904392 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1308) took 1015 ms. Trace:
I20260806 20:15:50.904699 30864 rpcz_store.cc:276] 0806 20:15:49.889223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.803802 (+914579us) service_pool.cc:227] Handling call
0806 20:15:50.904331 (+100529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:50.905977 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1559) took 2450 ms. Trace:
I20260806 20:15:50.868788 32530 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.906548 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:50.908782 32530 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:50.882856 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1568) took 2111 ms. Trace:
I20260806 20:15:50.900802 32640 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.910461 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1154) took 2784 ms. Trace:
I20260806 20:15:50.870558 32542 consensus_queue.cc:260] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.910874 30865 rpcz_store.cc:276] 0806 20:15:48.126339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.818556 (+1692217us) service_pool.cc:227] Handling call
0806 20:15:50.813574 (+995018us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:50.822852 (+ 9278us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:50.910265 (+ 87413us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b5400375350. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:50.910402 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2287551360}
I20260806 20:15:50.900478 32758 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:50.912556 32758 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.914670 32758 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.132867 32718 raft_consensus.cc:686] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:50.800151 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:50.807833 32576 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:50.918517 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1271) took 1036 ms. Trace:
I20260806 20:15:50.918337 32576 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.257371 32697 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:50.919541 32697 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.919055 30784 rpcz_store.cc:276] 0806 20:15:49.882158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.884438 (+ 2280us) service_pool.cc:227] Handling call
0806 20:15:50.918446 (+1034008us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:50.850235 30754 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.008s sys 0.007s
I20260806 20:15:50.920487 32576 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:50.920859 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 622298us
I20260806 20:15:50.853749 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:50.801956 32667 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:50.921806 32667 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.923746 32667 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.924345 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:50.924968 31421 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:50.854995 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:50.925815 30675 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.014s sys 0.015s
I20260806 20:15:50.925949 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1275) took 1031 ms. Trace:
I20260806 20:15:50.851560 30709 rpcz_store.cc:276] 0806 20:15:48.477971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.410022 (+932051us) service_pool.cc:227] Handling call
0806 20:15:49.428212 (+ 18190us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:49.428551 (+ 339us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:49.428581 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:49.429028 (+ 447us) log.cc:842] Serialized 26 byte log entry
0806 20:15:49.646559 (+217531us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:49.646605 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.646692 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:50.850960 (+1204268us) raft_consensus.cc:1671] finished
0806 20:15:50.850981 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:50.851124 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":30886,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":7838,"thread_start_us":215420,"threads_started":1,"wal-append.queue_time_us":391324}
W20260806 20:15:50.926441 30675 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.813s user 0.014s sys 0.015s
I20260806 20:15:50.918586 32718 consensus_queue.cc:237] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.926932 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:50.927579 30709 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:50.928022 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1174) took 1224 ms. Trace:
I20260806 20:15:50.928288 30709 rpcz_store.cc:276] 0806 20:15:49.703988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.926658 (+1222670us) service_pool.cc:227] Handling call
0806 20:15:50.927972 (+ 1314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:50.827167 32708 raft_consensus.cc:3032] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:50.929382 32708 raft_consensus.cc:729] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:50.929772 32712 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:50.929530 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:50.906374 30788 rpcz_store.cc:276] 0806 20:15:48.455144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.507869 (+ 52725us) service_pool.cc:227] Handling call
0806 20:15:48.508191 (+ 322us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:49.765517 (+1257326us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:50.905635 (+1140118us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003d4ad4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:50.905888 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2886172032}
I20260806 20:15:50.930270 32708 consensus_queue.cc:260] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.930240 32712 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.931924 32708 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:50.933629 32755 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:50.933725 32712 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.930805 30709 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:50.934283 32755 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.936733 32755 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:50.823845 32652 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.011s sys 0.017s
W20260806 20:15:50.861953 30711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1437) took 2638 ms (client timeout 1504 ms). Trace:
I20260806 20:15:50.822625 32750 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:50.837226 32611 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:50.842456 32507 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:50.937994 30711 rpcz_store.cc:269] 0806 20:15:48.220170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.227345 (+ 7175us) service_pool.cc:227] Handling call
0806 20:15:50.072869 (+1845524us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:50.858151 (+785282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5765504}
I20260806 20:15:50.938975 32507 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.939749 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:50.940100 32238 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:50.940323 30711 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:50.941007 32507 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:50.826563 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:50.863747 30866 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 115) took 2205 ms (client timeout 1810 ms). Trace:
I20260806 20:15:50.826799 32741 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:50.941756 30703 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
W20260806 20:15:50.942386 30703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1444) took 2472 ms (client timeout 1649 ms). Trace:
I20260806 20:15:50.942154 32741 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:50.942752 30703 rpcz_store.cc:269] 0806 20:15:48.469759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.505662 (+ 35903us) service_pool.cc:227] Handling call
0806 20:15:50.813141 (+2307479us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:50.942322 (+129181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13231872}
W20260806 20:15:50.944120 32652 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.830s user 0.012s sys 0.018s
I20260806 20:15:50.944305 32741 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:50.828562 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:50.827239 30713 rpcz_store.cc:276] 0806 20:15:48.417962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.432865 (+ 14903us) service_pool.cc:227] Handling call
0806 20:15:48.433161 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:49.349397 (+916236us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:50.288497 (+939100us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b5400030cd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:50.288717 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2100587392}
I20260806 20:15:50.938488 32750 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:50.865881 30871 rpcz_store.cc:269] 0806 20:15:48.120459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.151518 (+ 31059us) service_pool.cc:227] Handling call
0806 20:15:49.720547 (+1569029us) spinlock_profiling.cc:243] Waited 30.1 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:50.813536 (+1092989us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69127936}
I20260806 20:15:50.948267 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:50.851754 32580 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:50.947975 32750 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.909626 30783 rpcz_store.cc:276] 0806 20:15:48.771390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.186925 (+415535us) service_pool.cc:227] Handling call
0806 20:15:49.303083 (+116158us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:49.350363 (+ 47280us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:49.358007 (+ 7644us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:49.425831 (+ 67824us) log.cc:842] Serialized 26 byte log entry
0806 20:15:49.719759 (+293928us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b48001b0258. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71565167f1 00007f715650d8e7 00007f715650d77a 00007f715660195f 00007f7156601627 00007f7156615b2c 00007f7156615ad8 00007f7156615a58 00007f7156615a13 00007f7156614cbc 00007f7155596be6
0806 20:15:49.788751 (+ 68992us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:49.819699 (+ 30948us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.824299 (+ 4600us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:50.862132 (+1037833us) raft_consensus.cc:1671] finished
0806 20:15:50.865201 (+ 3069us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:50.882760 (+ 17559us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4790,"raft.run_cpu_time_us":2088,"raft.run_wall_time_us":370552,"spinlock_wait_cycles":9744256,"thread_start_us":346787,"threads_started":1,"wal-append.queue_time_us":470521}
I20260806 20:15:49.895350 32673 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.950408 32750 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:50.857887 32698 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:50.951483 32673 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.951542 32698 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:50.952790 32698 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:50.919023 32542 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:50.862978 32568 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.953655 32750 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.884160 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1147) took 2762 ms. Trace:
I20260806 20:15:50.911859 32640 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:50.905181 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:50.917210 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:50.072547 32443 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:50.919021 31422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 466) took 1219 ms. Trace:
W20260806 20:15:50.860968 30705 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.819s user 0.014s sys 0.022s
W20260806 20:15:50.827381 31415 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.010s sys 0.015s
I20260806 20:15:50.938757 32611 raft_consensus.cc:686] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:50.866231 30676 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:50.949389 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1309) took 1056 ms. Trace:
I20260806 20:15:50.962682 31422 rpcz_store.cc:276] 0806 20:15:49.700127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.700355 (+ 228us) service_pool.cc:227] Handling call
0806 20:15:49.700716 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:49.701080 (+ 364us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:49.701109 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:49.727630 (+ 26521us) log.cc:842] Serialized 26 byte log entry
0806 20:15:49.932354 (+204724us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:49.932394 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.932462 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:50.918819 (+986357us) raft_consensus.cc:1671] finished
0806 20:15:50.918839 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:50.918965 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7448,"raft.queue_time_us":1546,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":92,"thread_start_us":204407,"threads_started":1,"wal-append.queue_time_us":1120266}
W20260806 20:15:50.941682 30866 rpcz_store.cc:269] 0806 20:15:48.658623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.718508 (+ 59885us) service_pool.cc:227] Handling call
0806 20:15:50.863573 (+2145065us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b540019da54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:50.863691 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236267264}
W20260806 20:15:51.038138 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 172004us
I20260806 20:15:49.892601 32511 consensus_queue.cc:237] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.287602 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:50.963245 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1311) took 1058 ms. Trace:
I20260806 20:15:51.043171 30865 rpcz_store.cc:276] 0806 20:15:49.904976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.917009 (+1012033us) service_pool.cc:227] Handling call
0806 20:15:50.963187 (+ 46178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.044231 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:50.962200 30867 rpcz_store.cc:276] 0806 20:15:48.121441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.312106 (+1190665us) service_pool.cc:227] Handling call
0806 20:15:49.312412 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:49.479143 (+166731us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:49.479172 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:49.517199 (+ 38027us) log.cc:842] Serialized 26 byte log entry
0806 20:15:49.840846 (+323647us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:49.840890 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:49.840976 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:50.883942 (+1042966us) raft_consensus.cc:1671] finished
0806 20:15:50.883961 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:50.884101 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":202210,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":464434,"threads_started":2,"wal-append.queue_time_us":388828}
W20260806 20:15:50.857278 32635 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.258s user 0.024s sys 0.024s
I20260806 20:15:50.938208 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:50.870787 32574 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:50.955502 32568 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.037652 30871 rpcz_store.cc:276] 0806 20:15:49.893163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.948026 (+1054863us) service_pool.cc:227] Handling call
0806 20:15:50.949331 (+ 1305us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.051136 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1156) took 2923 ms. Trace:
I20260806 20:15:50.954665 30792 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:50.954936 30864 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:15:50.986653 32443 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:51.047042 30865 raft_consensus.cc:2396] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:51.038545 32743 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.048804 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.048969 30867 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:15:51.055308 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1310) took 1159 ms. Trace:
I20260806 20:15:51.051481 30871 rpcz_store.cc:276] 0806 20:15:48.127790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.050343 (+2922553us) service_pool.cc:227] Handling call
0806 20:15:51.050653 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.050840 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.050865 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.050905 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:51.050931 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.050980 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.051082 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.056350 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1305) took 1176 ms. Trace:
I20260806 20:15:51.056844 30866 rpcz_store.cc:276] 0806 20:15:49.880149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.038296 (+1158147us) service_pool.cc:227] Handling call
0806 20:15:51.056294 (+ 17998us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.056236 30865 rpcz_store.cc:276] 0806 20:15:49.895771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.043920 (+1148149us) service_pool.cc:227] Handling call
0806 20:15:51.055248 (+ 11328us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:51.047827 32635 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.451s user 0.024s sys 0.024s
I20260806 20:15:51.058602 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1160) took 2929 ms. Trace:
W20260806 20:15:51.040513 31415 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.956s user 0.011s sys 0.015s
I20260806 20:15:51.041237 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:51.062642 30861 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1298) took 2552 ms (client timeout 1910 ms). Trace:
W20260806 20:15:51.063099 30861 rpcz_store.cc:269] 0806 20:15:48.509887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.572669 (+ 62782us) service_pool.cc:227] Handling call
0806 20:15:51.062412 (+2489743us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54001e4454. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.062569 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":270982016}
I20260806 20:15:51.064843 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1168) took 2849 ms. Trace:
I20260806 20:15:51.065059 32611 consensus_queue.cc:237] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:51.047183 30674 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.819s user 0.000s sys 0.047s
W20260806 20:15:50.987037 30705 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.948s user 0.014s sys 0.022s
I20260806 20:15:50.926303 30785 rpcz_store.cc:276] 0806 20:15:49.894710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.828760 (+934050us) service_pool.cc:227] Handling call
0806 20:15:50.925891 (+ 97131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.056483 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1155) took 2929 ms. Trace:
I20260806 20:15:51.068578 30867 rpcz_store.cc:276] 0806 20:15:48.127516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.048356 (+2920840us) service_pool.cc:227] Handling call
0806 20:15:51.048738 (+ 382us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:51.056193 (+ 7455us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.056418 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.070245 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1295) took 2757 ms. Trace:
I20260806 20:15:51.060912 30871 rpcz_store.cc:276] 0806 20:15:48.128782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.056880 (+2928098us) service_pool.cc:227] Handling call
0806 20:15:51.057215 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.057387 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.057412 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.057474 (+ 62us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:51.057503 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.057554 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.057662 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.053982 32647 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:51.067549 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 144718us
I20260806 20:15:51.070597 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1167) took 2876 ms. Trace:
I20260806 20:15:50.966198 32713 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.086076 30865 rpcz_store.cc:276] 0806 20:15:48.183102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.058891 (+2875789us) service_pool.cc:227] Handling call
0806 20:15:51.059193 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.059353 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.059376 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.059412 (+ 36us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:51.059435 (+ 23us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.059484 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.059574 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.087863 30865 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 20:15:51.084966 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 43837.4us
I20260806 20:15:51.070533 30867 rpcz_store.cc:276] 0806 20:15:48.313037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.069319 (+2756282us) service_pool.cc:227] Handling call
0806 20:15:51.069804 (+ 485us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.069960 (+ 156us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.069982 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.070019 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:51.070048 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.070095 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.070194 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.089664 30676 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:15:51.067739 30674 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.840s user 0.000s sys 0.047s
I20260806 20:15:51.076467 30711 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:51.096043 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1175) took 1378 ms. Trace:
I20260806 20:15:51.054545 32751 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:51.098162 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1296) took 2720 ms. Trace:
I20260806 20:15:51.127135 30865 rpcz_store.cc:276] 0806 20:15:48.378924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.087161 (+2708237us) service_pool.cc:227] Handling call
0806 20:15:51.087511 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.088205 (+ 694us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.098103 (+ 9898us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.128612 30865 raft_consensus.cc:1226] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260806 20:15:51.100934 32678 log.cc:924] Time spent T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 2.477s user 0.009s sys 0.054s
I20260806 20:15:51.096508 30866 raft_consensus.cc:1261] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260806 20:15:51.129941 30866 pending_rounds.cc:76] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 1
I20260806 20:15:51.130177 30866 pending_rounds.cc:97] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:15:51.130110 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.130599 30866 raft_consensus.cc:2866] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:51.131501 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1162) took 3002 ms. Trace:
I20260806 20:15:51.131804 30866 rpcz_store.cc:276] 0806 20:15:48.129264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.057505 (+2928241us) service_pool.cc:227] Handling call
0806 20:15:51.072694 (+ 15189us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.131265 (+ 58571us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.131447 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.107122 32078 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:51.053305 32576 consensus_queue.cc:1046] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20260806 20:15:51.112947 32632 raft_consensus.cc:3032] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 20:15:51.116847 30710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1176) took 1398 ms (client timeout 1769 ms). Trace:
I20260806 20:15:51.119831 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 146) took 2572 ms. Trace:
I20260806 20:15:51.123386 32580 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.125638 30757 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:51.129256 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1174) took 2663 ms. Trace:
I20260806 20:15:51.065213 30861 rpcz_store.cc:276] 0806 20:15:48.215206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.063953 (+2848747us) service_pool.cc:227] Handling call
0806 20:15:51.064310 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.064502 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.064530 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.064574 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.064614 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.064669 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.064788 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:51.137048 30710 rpcz_store.cc:269] 0806 20:15:49.718074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.718515 (+ 441us) service_pool.cc:227] Handling call
0806 20:15:51.116649 (+1398134us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54000906d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.116759 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":326820736}
I20260806 20:15:51.137583 30865 rpcz_store.cc:276] 0806 20:15:48.465583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.128011 (+2662428us) service_pool.cc:227] Handling call
0806 20:15:51.128381 (+ 370us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:51.129001 (+ 620us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.129204 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.096421 30711 rpcz_store.cc:276] 0806 20:15:49.717721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.939495 (+1221774us) service_pool.cc:227] Handling call
0806 20:15:51.095979 (+156484us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.137240 30703 rpcz_store.cc:276] 0806 20:15:48.546986 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.943512 (+2396526us) service_pool.cc:227] Handling call
0806 20:15:51.118713 (+175201us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.118877 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.118899 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.118936 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:51.118962 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.119013 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.119677 (+ 664us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54000906d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.119777 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402207104}
I20260806 20:15:51.138447 30861 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:51.138381 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:51.122745 30834 connection.cc:232] RPC call timeout handler was delayed by -1.8306869149964768s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
I20260806 20:15:51.067744 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1563) took 2558 ms. Trace:
W20260806 20:15:51.139740 30673 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.909s user 0.000s sys 0.012s
I20260806 20:15:51.139194 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:51.067159 31418 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 325) took 2294 ms (client timeout 1916 ms). Trace:
I20260806 20:15:51.143697 30703 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:51.142416 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:51.144279 30673 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.914s user 0.000s sys 0.013s
I20260806 20:15:51.144361 30711 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:51.141817 30754 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:51.070703 30709 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:51.060869 32743 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:51.088311 32697 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.096153 32430 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:51.077781 32238 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:51.149644 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:51.151109 30834 connection.cc:232] RPC call timeout handler was delayed by -1.2861869150074199s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
W20260806 20:15:51.164398 30834 connection.cc:232] RPC call timeout handler was delayed by -1.1518167220056057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16490000000000002s for a response
I20260806 20:15:51.150718 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.166652 30834 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.165623 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.145388 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1173) took 1441 ms. Trace:
I20260806 20:15:51.167704 30709 rpcz_store.cc:276] 0806 20:15:49.703725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.929269 (+1225544us) service_pool.cc:227] Handling call
0806 20:15:51.145333 (+216064us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.169045 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.077586 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 145) took 2554 ms. Trace:
I20260806 20:15:51.170358 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:51.170934 30710 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:15:51.173384 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 356574us
W20260806 20:15:51.175428 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 49883.4us
I20260806 20:15:51.176194 30757 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:51.177102 31389 leader_election.cc:334] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.527s (SENT)
W20260806 20:15:51.177421 30676 connection.cc:232] RPC call timeout handler was delayed by -1.2563035990169737s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
I20260806 20:15:51.178535 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.136729 32755 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:51.140971 32632 raft_consensus.cc:729] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:51.179183 30703 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:51.139220 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 114) took 2635 ms. Trace:
I20260806 20:15:51.157145 32511 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:51.193280 30861 rpcz_store.cc:276] 0806 20:15:48.504031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.137742 (+2633711us) service_pool.cc:227] Handling call
0806 20:15:51.138078 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.138961 (+ 883us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.139166 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.144848 32580 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.194561 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.194705 32758 consensus_queue.cc:1035] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:51.139807 32735 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:51.195379 32580 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:51.153017 32680 consensus_queue.cc:1046] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260806 20:15:51.134342 32757 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:51.144671 32530 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.181156 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:51.157590 32741 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.098941 32647 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.182703 31415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 470) took 1290 ms. Trace:
I20260806 20:15:51.190708 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.191151 32765 raft_consensus.cc:3032] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:51.162313 32507 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.192606 32756 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:51.184386 32707 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.195803 32735 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.150938 32652 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.197630 30676 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.178377 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.153849 32635 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:51.215472 32611 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:51.184134 32697 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.059s
I20260806 20:15:51.150000 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.201268 30756 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:51.149238 32574 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.206768 30710 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:51.275175 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:51.276428 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.155679 32743 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.277093 30710 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
W20260806 20:15:51.278816 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 81764.3us
I20260806 20:15:51.210400 30709 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:51.212307 30832 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.207466 32672 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:51.226258 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1173) took 2782 ms. Trace:
I20260806 20:15:51.190029 32763 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:51.209410 32616 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260806 20:15:51.200295 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.180613 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.255762 32751 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:51.236986 32530 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.182257 32673 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:51.198105 32750 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:51.143242 30786 rpcz_store.cc:276] 0806 20:15:48.509190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.583259 (+ 74069us) service_pool.cc:227] Handling call
0806 20:15:48.587973 (+ 4714us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:49.473279 (+885306us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.054300 (+1581021us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54001a0754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:51.060404 (+ 6104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1946305280}
I20260806 20:15:51.247493 32507 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.207592 32621 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:51.251480 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 53951.5us
I20260806 20:15:51.182893 32712 consensus_queue.cc:1035] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
W20260806 20:15:51.157105 31418 rpcz_store.cc:269] 0806 20:15:48.772385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.774250 (+ 1865us) service_pool.cc:227] Handling call
0806 20:15:50.287399 (+1513149us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:51.067041 (+779642us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11269376}
I20260806 20:15:51.170634 32238 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.198096 30834 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.258832 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.208335 32668 consensus_queue.cc:1046] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260806 20:15:51.259413 32713 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.236780 32757 raft_consensus.cc:686] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:51.259914 32764 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:51.261521 32735 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.264973 32647 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:51.266252 32761 log.cc:924] Time spent T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260806 20:15:51.259428 32611 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.328665 32611 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:51.265579 32767 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:15:51.328500 32757 consensus_queue.cc:237] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.244668 32765 raft_consensus.cc:729] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:51.196996 32755 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.279003 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.290740 32680 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:51.280628 30867 rpcz_store.cc:276] 0806 20:15:48.443895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.090281 (+2646386us) service_pool.cc:227] Handling call
0806 20:15:51.090518 (+ 237us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.090664 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:51.147216 (+ 56552us) log.cc:842] Serialized 14 byte log entry
0806 20:15:51.225858 (+ 78642us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.225916 (+ 58us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:51.225959 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.226052 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.226195 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15436,"prepare.queue_time_us":147378,"prepare.run_wall_time_us":43,"raft.queue_time_us":21626,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":6427,"spinlock_wait_cycles":23528576,"thread_start_us":78646,"threads_started":2}
I20260806 20:15:51.332664 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.333371 32530 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:51.333863 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.334910 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.335619 30867 raft_consensus.cc:2396] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:51.336300 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.282775 30711 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:15:51.337742 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1175) took 2569 ms. Trace:
I20260806 20:15:51.337854 32507 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:51.338167 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:51.339617 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.312249 32621 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.341017 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.284446 32672 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.342705 32621 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.343076 32664 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:51.343544 32755 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.344522 32765 consensus_queue.cc:260] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.345961 32765 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:51.238933 30703 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:51.348539 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.349164 30703 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:51.349967 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.350554 30703 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
W20260806 20:15:51.351163 32762 log.cc:924] Time spent T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20260806 20:15:51.351397 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.352531 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.180101 30705 rpcz_store.cc:276] 0806 20:15:48.523659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.510641 (+986982us) service_pool.cc:227] Handling call
0806 20:15:49.511009 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:49.511229 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:15:49.847097 (+335868us) log.cc:842] Serialized 14 byte log entry
0806 20:15:51.077159 (+1230062us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.077224 (+ 65us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:15:51.077276 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.077356 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.077518 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24289,"prepare.queue_time_us":365480,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":3284,"raft.queue_time_us":291235,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":98,"thread_start_us":1482646,"threads_started":3,"wal-append.queue_time_us":946851}
I20260806 20:15:51.353582 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:51.349836 32680 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.354342 30705 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:51.354745 32672 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:51.362461 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.184329 32718 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:51.282207 32763 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:51.288738 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1171) took 2936 ms. Trace:
I20260806 20:15:51.305621 30709 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:51.286252 32443 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.304188 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.238546 31415 rpcz_store.cc:276] 0806 20:15:49.888995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.890647 (+ 1652us) service_pool.cc:227] Handling call
0806 20:15:50.009608 (+118961us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:50.043144 (+ 33536us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:15:51.078757 (+1035613us) log.cc:842] Serialized 14 byte log entry
0806 20:15:51.133963 (+ 55206us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.135997 (+ 2034us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:15:51.143878 (+ 7881us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.162612 (+ 18734us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.178938 (+ 16326us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":999310,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":7273,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":6625,"thread_start_us":1028682,"threads_started":2,"wal-append.queue_time_us":112516}
I20260806 20:15:51.281936 32574 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:51.256047 32632 consensus_queue.cc:260] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.254211 32707 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.374409 32618 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:51.374770 32618 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:51.375523 32707 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:51.256482 32652 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.320655 31423 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:51.326213 30861 raft_consensus.cc:2396] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:51.377121 32574 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:51.377592 32574 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:51.377796 32652 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:51.379453 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1151) took 3257 ms. Trace:
I20260806 20:15:51.325691 32764 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:51.269511 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 68321.3us
I20260806 20:15:51.207396 32511 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.380985 30866 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:51.330638 30710 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:51.381493 30756 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:51.382073 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.355206 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.355832 32750 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:51.235514 32741 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:51.356630 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 158612us
I20260806 20:15:51.357191 32712 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:51.352478 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1152) took 2803 ms. Trace:
I20260806 20:15:51.391644 31423 raft_consensus.cc:1226] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:51.390903 30834 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.390533 30754 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:51.393639 30834 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:15:51.353610 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 141420us
I20260806 20:15:51.391438 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.338114 30865 rpcz_store.cc:276] 0806 20:15:48.768204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.138900 (+2370696us) service_pool.cc:227] Handling call
0806 20:15:51.139208 (+ 308us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:51.140625 (+ 1417us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.140652 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:51.141074 (+ 422us) log.cc:842] Serialized 26 byte log entry
0806 20:15:51.255362 (+114288us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:51.255403 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.255468 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:51.337533 (+ 82065us) raft_consensus.cc:1671] finished
0806 20:15:51.337551 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.337686 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11039,"raft.queue_time_us":1103,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":18663,"thread_start_us":113981,"threads_started":1,"wal-append.queue_time_us":114325}
I20260806 20:15:51.360236 30711 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:51.369204 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.364154 32718 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:51.395133 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 217051us
I20260806 20:15:51.357508 32668 consensus_queue.cc:1225] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer 42cabca48db7403b8523a3382a046043 log is divergent from this leader: its last log entry 2.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 20:15:51.361806 30867 raft_consensus.cc:1261] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:51.364974 30871 rpcz_store.cc:276] 0806 20:15:48.352425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.071718 (+2719293us) service_pool.cc:227] Handling call
0806 20:15:51.072024 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:51.086254 (+ 14230us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.086289 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:51.086752 (+ 463us) log.cc:842] Serialized 26 byte log entry
0806 20:15:51.238844 (+152092us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.238881 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.238953 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:51.288505 (+ 49552us) raft_consensus.cc:1671] finished
0806 20:15:51.288527 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.288664 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24129,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":35211,"thread_start_us":151797,"threads_started":1,"wal-append.queue_time_us":167193}
I20260806 20:15:51.396454 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:51.371850 30712 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1158) took 2814 ms (client timeout 2059 ms). Trace:
W20260806 20:15:51.397691 30712 rpcz_store.cc:269] 0806 20:15:48.557277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.918138 (+360861us) service_pool.cc:227] Handling call
0806 20:15:51.371638 (+2453500us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x7b54000c8354. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.371778 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162900352}
I20260806 20:15:51.370572 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.385576 32764 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.380647 32610 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.386130 32443 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:51.364743 32763 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:51.388617 30710 raft_consensus.cc:2396] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:51.388468 30756 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:51.319977 313 consensus_queue.cc:1046] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260806 20:15:51.392419 30713 rpcz_store.cc:276] 0806 20:15:48.548874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.945613 (+2396739us) service_pool.cc:227] Handling call
0806 20:15:50.945988 (+ 375us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:50.946368 (+ 380us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:50.946396 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:50.946930 (+ 534us) log.cc:842] Serialized 52 byte log entry
0806 20:15:51.040229 (+ 93299us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.040548 (+ 319us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.040620 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:51.352277 (+311657us) raft_consensus.cc:1671] finished
0806 20:15:51.352298 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.352423 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1274,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"thread_start_us":92903,"threads_started":1,"wal-append.queue_time_us":180090}
I20260806 20:15:51.357204 32680 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:51.369904 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 65797.6us
I20260806 20:15:51.379819 30869 rpcz_store.cc:276] 0806 20:15:48.122443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.335966 (+1213523us) service_pool.cc:227] Handling call
0806 20:15:50.863721 (+1527755us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:50.863912 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:50.863938 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:50.863976 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:50.864002 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:50.864056 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.379290 (+515234us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b540019da50. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.379397 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3512764160}
I20260806 20:15:51.395859 32712 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:51.389086 32672 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:51.402539 30834 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:51.403215 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.391721 32758 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:51.387179 32647 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:51.396445 32744 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:51.398825 32718 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:51.399488 30703 raft_consensus.cc:2379] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:51.407833 30756 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:51.398221 32673 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:51.414835 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.409937 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.413854 32568 raft_consensus.cc:3032] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:51.422883 30676 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.423379 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1160) took 2865 ms. Trace:
I20260806 20:15:51.423758 30703 rpcz_store.cc:276] 0806 20:15:48.558191 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.421729 (+2863538us) service_pool.cc:227] Handling call
0806 20:15:51.422920 (+ 1191us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.423081 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.423104 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.423141 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:51.423166 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.423228 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.423326 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:51.414496 32685 log.cc:924] Time spent T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 2.757s user 0.001s sys 0.023s
I20260806 20:15:51.398526 314 consensus_queue.cc:1046] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260806 20:15:51.405220 32668 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:51.418810 30711 raft_consensus.cc:2396] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:51.418938 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.420672 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.410964 30705 raft_consensus.cc:2396] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:15:51.409488 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.416199 32758 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:51.408649 32642 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:51.415481 32712 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:51.414666 32707 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:51.425674 30676 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.426652 30756 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:51.437726 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.438182 30756 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:51.438334 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.439507 30756 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:51.419505 32744 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:51.440590 30832 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.405359 32610 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.430734 30834 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:51.432101 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.424834 32565 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:15:51.434041 30870 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1293) took 3349 ms (client timeout 2035 ms). Trace:
I20260806 20:15:51.421252 32673 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:51.421973 32443 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:51.424998 32568 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:51.426258 316 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:51.428845 32755 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:51.430353 32668 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:51.444271 32744 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:51.441349 32707 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.448287 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.440582 32621 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:51.426975 31420 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:51.446020 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.451429 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:51.445266 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:15:51.571431 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 126221us
I20260806 20:15:51.572325 30834 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:51.434355 32758 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:51.450402 30870 rpcz_store.cc:269] 0806 20:15:48.084596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.084726 (+ 130us) service_pool.cc:227] Handling call
0806 20:15:51.432511 (+3347785us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540007dd54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.433962 (+ 1451us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3351636736}
I20260806 20:15:51.432632 32652 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:51.421383 32647 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.454034 32443 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:51.454190 32673 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:51.447654 32642 raft_consensus.cc:686] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:51.544147 32632 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:51.444980 318 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:15:51.546885 30706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1155) took 2990 ms (client timeout 1550 ms). Trace:
I20260806 20:15:51.451050 32616 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:51.547776 32668 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:51.570026 32741 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:51.548959 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:50.341637 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:15:51.551133 32610 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:51.551721 31414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 348) took 1116 ms. Trace:
I20260806 20:15:51.449836 32750 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:51.447336 32274 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:51.446009 32680 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:15:51.555650 30792 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1576) took 1851 ms (client timeout 1728 ms). Trace:
W20260806 20:15:51.628095 30792 rpcz_store.cc:269] 0806 20:15:49.704397 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.704520 (+ 123us) service_pool.cc:227] Handling call
0806 20:15:51.491706 (+1787186us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000573d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.555581 (+ 63875us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2857007488}
I20260806 20:15:51.548046 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1297) took 3064 ms. Trace:
I20260806 20:15:51.629175 30866 rpcz_store.cc:276] 0806 20:15:48.483706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.132483 (+2648777us) service_pool.cc:227] Handling call
0806 20:15:51.158780 (+ 26297us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.381354 (+222574us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.492595 (+111241us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400149e54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:51.547987 (+ 55392us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":510229888}
I20260806 20:15:51.565672 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.561221 30867 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:51.573278 30834 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:51.598901 30832 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:51.575670 30790 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:51.570405 32078 raft_consensus.cc:3032] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:51.633383 32078 raft_consensus.cc:729] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:51.633584 32767 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:51.634099 32767 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.634296 32763 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.603121 32426 consensus_queue.cc:1046] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260806 20:15:51.636483 31422 raft_consensus.cc:2379] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
W20260806 20:15:51.607877 30706 rpcz_store.cc:269] 0806 20:15:48.556257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.623247 (+ 66990us) service_pool.cc:227] Handling call
0806 20:15:51.482750 (+2859503us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b54000c7954. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.546812 (+ 64062us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1985189248}
I20260806 20:15:51.554034 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.570952 316 raft_consensus.cc:686] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:51.548900 32618 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:51.642928 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 44116us
W20260806 20:15:51.587529 32671 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.102s user 0.010s sys 0.012s
I20260806 20:15:51.548874 32640 raft_consensus.cc:3032] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:51.597754 32530 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:51.598170 32708 raft_consensus.cc:2781] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 20:15:51.575985 30864 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1306) took 1694 ms (client timeout 2091 ms). Trace:
I20260806 20:15:51.654306 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1164) took 3057 ms. Trace:
I20260806 20:15:51.671006 30706 rpcz_store.cc:276] 0806 20:15:48.596688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.637904 (+3041216us) service_pool.cc:227] Handling call
0806 20:15:51.653844 (+ 15940us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.654012 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.654034 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.654073 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.654099 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.654147 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.654252 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:15:51.620512 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 182611us
I20260806 20:15:51.597334 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1145) took 3455 ms. Trace:
I20260806 20:15:51.599527 32652 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:51.655520 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1161) took 3097 ms. Trace:
I20260806 20:15:51.600677 32443 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:51.606328 32013 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.603395 32672 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.602360 32511 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:51.652145 32671 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.164s user 0.012s sys 0.013s
I20260806 20:15:51.632056 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1578) took 1723 ms. Trace:
I20260806 20:15:51.680927 30790 rpcz_store.cc:276] 0806 20:15:49.908637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.574006 (+1665369us) service_pool.cc:227] Handling call
0806 20:15:51.575503 (+ 1497us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.631778 (+ 56275us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.631991 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:51.607664 318 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:51.639009 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.616631 32756 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.598645 32507 consensus_queue.cc:1046] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260806 20:15:51.598498 32510 consensus_queue.cc:1046] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
W20260806 20:15:51.643806 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 180059us
I20260806 20:15:51.647521 30710 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:51.649989 32426 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:15:51.650599 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 77395.6us
I20260806 20:15:51.707455 30834 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:51.707732 30707 raft_consensus.cc:1226] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:51.636386 32767 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:51.603293 32574 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.628527 32274 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:51.673166 30676 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:51.600899 32621 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.628043 32680 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
W20260806 20:15:51.670109 30864 rpcz_store.cc:269] 0806 20:15:49.880488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.906101 (+1025613us) service_pool.cc:227] Handling call
0806 20:15:51.504582 (+598481us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b54002fa8d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.574660 (+ 70078us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108567296}
W20260806 20:15:51.649902 310 log.cc:924] Time spent T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260806 20:15:51.704731 32756 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.682195 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.668602 32530 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:51.652361 32640 raft_consensus.cc:729] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:51.706617 32345 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.642827 32750 raft_consensus.cc:686] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:51.668601 32610 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:51.685751 31421 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:51.657579 32763 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:51.686430 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:51.654207 32744 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.678615 32652 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:51.670747 32708 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:51.683179 32672 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.681042 32013 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.692404 32649 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:51.693173 32238 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:51.695127 32712 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.699793 31419 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:51.680135 313 raft_consensus.cc:1050] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:51.614786 32568 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.678696 31416 raft_consensus.cc:2396] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:15:51.634312 32078 consensus_queue.cc:260] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.719681 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:51.724979 32621 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.725803 32680 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:51.653286 32667 raft_consensus.cc:3032] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:51.731273 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1153) took 3177 ms. Trace:
I20260806 20:15:51.597543 30861 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:51.731563 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1159) took 3173 ms. Trace:
W20260806 20:15:51.738253 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 65147.1us
I20260806 20:15:51.740886 30706 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:51.747820 30834 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:51.697862 318 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:51.710846 32426 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:51.738626 32274 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:51.670415 32642 consensus_queue.cc:237] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.749634 32530 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:51.744763 32665 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:51.624421 31414 rpcz_store.cc:276] 0806 20:15:50.435864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.452094 (+ 16230us) service_pool.cc:227] Handling call
0806 20:15:50.666998 (+214904us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.149409 (+482411us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.149468 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.149518 (+ 50us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:51.149549 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.485968 (+336419us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b54005eea54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:51.551528 (+ 65560us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.551660 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1095097728}
I20260806 20:15:51.753544 32616 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:51.781271 32610 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:51.782043 32345 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:51.844806 30708 raft_consensus.cc:2396] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:51.795926 32708 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:51.843600 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:51.848018 30787 raft_consensus.cc:3032] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:51.949934 30787 raft_consensus.cc:729] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:51.846884 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:51.750613 32718 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:51.838578 330 log.cc:924] Time spent T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260806 20:15:51.868434 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:51.868801 30712 rpcz_store.cc:276] 0806 20:15:48.554095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.403071 (+2848976us) service_pool.cc:227] Handling call
0806 20:15:51.403405 (+ 334us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:51.403730 (+ 325us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.403757 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:51.408287 (+ 4530us) log.cc:842] Serialized 52 byte log entry
0806 20:15:51.624192 (+215905us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.624228 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.624294 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:51.731025 (+106731us) raft_consensus.cc:1671] finished
0806 20:15:51.731046 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.731211 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":455,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":3626,"thread_start_us":196107,"threads_started":1,"wal-append.queue_time_us":250472}
I20260806 20:15:51.958783 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.838428 32013 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.849017 32647 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.815130 32507 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.894821 31388 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:51.853075 32744 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:51.851982 313 raft_consensus.cc:1054] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Unable to promote non-voter 58f2f6256ee14cb38a1dbcbf5bb590a0: Illegal state: Leader has not yet committed an operation in its own term
I20260806 20:15:51.760496 316 consensus_queue.cc:237] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.893242 30713 rpcz_store.cc:276] 0806 20:15:48.557889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.406915 (+2849026us) service_pool.cc:227] Handling call
0806 20:15:51.417288 (+ 10373us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:51.417616 (+ 328us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.417645 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:51.425867 (+ 8222us) log.cc:842] Serialized 26 byte log entry
0806 20:15:51.625116 (+199249us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.625156 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.625225 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:51.731356 (+106131us) raft_consensus.cc:1671] finished
0806 20:15:51.731375 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.731508 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":399,"raft.queue_time_us":266,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":2865,"thread_start_us":179067,"threads_started":1,"wal-append.queue_time_us":204387}
I20260806 20:15:52.030110 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:52.030840 30713 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:52.031755 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:52.032421 30713 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:51.914635 30676 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:51.905290 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 157543us
I20260806 20:15:52.035434 30834 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:51.921504 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.868439 32238 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:51.673988 30863 rpcz_store.cc:276] 0806 20:15:48.120970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:48.784956 (+663986us) service_pool.cc:227] Handling call
0806 20:15:51.065800 (+2280844us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.066171 (+ 371us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.066203 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.066247 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.066277 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.066337 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.505313 (+438976us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b54001e4450. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:51.575901 (+ 70588us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":137,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":91,"spinlock_wait_cycles":5244661504}
I20260806 20:15:51.868276 32667 raft_consensus.cc:729] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:51.892513 32568 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:51.846560 32635 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260806 20:15:51.951824 30790 raft_consensus.cc:2379] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:51.903307 32618 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.953472 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:51.891819 32707 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:51.904740 32426 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:51.950809 30787 consensus_queue.cc:260] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.799687 32697 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260806 20:15:51.934937 32610 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:51.890677 32750 consensus_queue.cc:237] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:51.926607 32763 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:52.078276 30787 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:52.057902 32647 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:51.909977 32671 consensus_queue.cc:1035] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20260806 20:15:51.953935 32717 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:52.024066 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:51.970005 32672 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.033270 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:51.933072 32370 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.083s
I20260806 20:15:52.038232 30706 raft_consensus.cc:2379] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
W20260806 20:15:52.038195 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 123664us
I20260806 20:15:52.197759 30676 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:52.047165 30834 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:52.199038 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 151983us
W20260806 20:15:52.038537 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 143823us
I20260806 20:15:52.204128 30713 raft_consensus.cc:2379] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:51.965865 30783 raft_consensus.cc:1261] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:51.954372 32510 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:51.960325 30784 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:51.901721 32640 consensus_queue.cc:260] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.223191 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.074442 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.224926 30790 raft_consensus.cc:2379] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:52.225741 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:52.080081 333 log.cc:924] Time spent T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20260806 20:15:51.966820 32756 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:51.959417 30712 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:52.101928 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:51.934876 32078 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:52.029392 32574 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.066139 32757 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:52.108582 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.060355 32578 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:52.274212 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.275470 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.276239 30791 raft_consensus.cc:2396] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:15:52.188234 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1151) took 3566 ms. Trace:
I20260806 20:15:52.277864 30707 rpcz_store.cc:276] 0806 20:15:48.546714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.554921 (+1008207us) service_pool.cc:227] Handling call
0806 20:15:49.563706 (+ 8785us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.708081 (+2144375us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.112413 (+404332us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b54000a91d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:52.112606 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4930421632}
I20260806 20:15:51.974713 32013 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:51.968544 32443 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.213183 30865 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 20:15:52.213739 30676 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:52.318809 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 105155us
I20260806 20:15:52.214584 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:52.021718 340 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:52.219508 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 309223us
I20260806 20:15:52.115648 32647 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:52.243328 32621 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.246732 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:52.188201 32744 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:52.220650 32717 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.073524 32238 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.231633 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:52.228323 332 log.cc:924] Time spent T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.437s user 0.001s sys 0.000s
I20260806 20:15:52.328202 32672 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:52.328651 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:52.277573 321 log.cc:924] Time spent T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.345s user 0.001s sys 0.000s
I20260806 20:15:52.220206 32712 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:52.354117 32673 consensus_meta.cc:287] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.747s user 0.003s sys 0.008s
I20260806 20:15:52.277108 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.373751 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.029740 32507 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.226925 32638 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 20:15:52.301841 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 352971us
I20260806 20:15:52.323036 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 148) took 3452 ms. Trace:
I20260806 20:15:52.388729 30713 rpcz_store.cc:276] 0806 20:15:48.870652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.322221 (+3451569us) service_pool.cc:227] Handling call
0806 20:15:52.322547 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.322725 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:52.322761 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:52.322805 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:52.322834 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.322883 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:52.322983 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.351609 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:52.390635 30713 raft_consensus.cc:1261] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:15:52.391247 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1170) took 3461 ms. Trace:
I20260806 20:15:52.391599 30713 rpcz_store.cc:276] 0806 20:15:48.930146 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.389916 (+3459770us) service_pool.cc:227] Handling call
0806 20:15:52.390295 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.391005 (+ 710us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.391196 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.273382 32757 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:52.336889 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:52.242033 32510 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:52.108642 32763 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.399691 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 150) took 2762 ms. Trace:
I20260806 20:15:52.231634 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:52.278386 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.246721 30866 raft_consensus.cc:1226] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 20:15:52.245070 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.358199 30790 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:52.428253 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.428954 30790 raft_consensus.cc:2379] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:52.270785 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.342588 32647 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:15:52.350996 333 log.cc:924] Time spent T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260806 20:15:52.364876 32672 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:52.363353 32345 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:52.360615 32758 consensus_meta.cc:287] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.753s user 0.009s sys 0.002s
I20260806 20:15:52.359091 32718 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:52.340838 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.421597 30713 rpcz_store.cc:276] 0806 20:15:49.637380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.398869 (+2761489us) service_pool.cc:227] Handling call
0806 20:15:52.399215 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.399378 (+ 163us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:52.399405 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:52.399445 (+ 40us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:15:52.399473 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.399525 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:52.399631 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.386917 32638 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:52.429419 30785 raft_consensus.cc:2379] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:52.364639 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.487416 32638 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:52.271763 30712 raft_consensus.cc:1261] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:52.487632 32238 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:52.448884 30706 raft_consensus.cc:1226] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:52.488281 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1165) took 3883 ms. Trace:
I20260806 20:15:52.354959 32744 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.488638 30712 rpcz_store.cc:276] 0806 20:15:48.605033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.269951 (+3664918us) service_pool.cc:227] Handling call
0806 20:15:52.271448 (+ 1497us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.487999 (+216551us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.488225 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.490235 32744 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:52.342867 340 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.491606 32763 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:52.492138 340 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:51.677591 30703 rpcz_store.cc:276] 0806 20:15:48.558558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.424445 (+2865887us) service_pool.cc:227] Handling call
0806 20:15:51.431539 (+ 7094us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.431699 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:15:51.614575 (+182876us) log.cc:842] Serialized 14 byte log entry
0806 20:15:51.655002 (+ 40427us) log.cc:842] Serialized 14 byte log entry
0806 20:15:51.655185 (+ 183us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:51.655226 (+ 41us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:51.655269 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.655337 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:51.655456 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11425,"raft.queue_time_us":43597,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":6861,"thread_start_us":199892,"threads_started":2}
I20260806 20:15:52.493729 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1172) took 2790 ms. Trace:
I20260806 20:15:52.494053 30703 rpcz_store.cc:276] 0806 20:15:49.703444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.492890 (+2789446us) service_pool.cc:227] Handling call
0806 20:15:52.493230 (+ 340us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.493399 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:52.493426 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:52.493483 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:52.493510 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.493558 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:52.493673 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.453276 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.330150 32013 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.496237 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.496966 32013 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:52.456323 30708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1448) took 2816 ms (client timeout 2082 ms). Trace:
W20260806 20:15:52.497601 30708 rpcz_store.cc:269] 0806 20:15:49.640344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:50.853474 (+1213130us) service_pool.cc:227] Handling call
0806 20:15:52.453512 (+1600038us) spinlock_profiling.cc:243] Waited 923 ms on lock 0x7b54000d5f54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:52.456257 (+ 2745us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2122174208}
I20260806 20:15:52.488576 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1447) took 2848 ms. Trace:
I20260806 20:15:52.498777 30706 rpcz_store.cc:276] 0806 20:15:49.640029 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.426464 (+2786435us) service_pool.cc:227] Handling call
0806 20:15:52.448728 (+ 22264us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.488337 (+ 39609us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.488525 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.499272 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 152) took 1460 ms. Trace:
I20260806 20:15:52.454969 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.390941 32712 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:52.500136 30706 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:52.500726 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 153) took 1450 ms. Trace:
I20260806 20:15:52.500607 32712 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.499636 30708 rpcz_store.cc:276] 0806 20:15:51.039011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.498549 (+1459538us) service_pool.cc:227] Handling call
0806 20:15:52.498836 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.499000 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:52.499024 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:52.499062 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:15:52.499089 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.499134 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:52.499223 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.502586 32712 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:52.502803 30708 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:52.478438 30713 raft_consensus.cc:1226] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:15:52.503501 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1450) took 2862 ms. Trace:
I20260806 20:15:52.503806 30713 rpcz_store.cc:276] 0806 20:15:49.640947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.455176 (+2814229us) service_pool.cc:227] Handling call
0806 20:15:52.455481 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:52.503259 (+ 47778us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.503452 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.504980 30713 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 20:15:52.505523 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1457) took 1389 ms. Trace:
I20260806 20:15:52.505802 30713 rpcz_store.cc:276] 0806 20:15:51.116510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.504471 (+1387961us) service_pool.cc:227] Handling call
0806 20:15:52.504776 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:52.505295 (+ 519us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.505476 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.507243 30713 raft_consensus.cc:1261] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:15:52.501042 30706 rpcz_store.cc:276] 0806 20:15:51.050120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.499498 (+1449378us) service_pool.cc:227] Handling call
0806 20:15:52.499806 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.500507 (+ 701us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.500677 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.507763 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1178) took 1317 ms. Trace:
I20260806 20:15:52.478735 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:52.480295 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1156) took 3923 ms. Trace:
I20260806 20:15:52.509753 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1452) took 1394 ms. Trace:
W20260806 20:15:52.485163 32652 consensus_meta.cc:287] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.681s user 0.007s sys 0.000s
I20260806 20:15:52.508080 30713 rpcz_store.cc:276] 0806 20:15:51.189998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.506587 (+1316589us) service_pool.cc:227] Handling call
0806 20:15:52.506962 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.507537 (+ 575us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.507717 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.510320 30709 rpcz_store.cc:276] 0806 20:15:48.556598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.404566 (+2847968us) service_pool.cc:227] Handling call
0806 20:15:51.415354 (+ 10788us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:51.415694 (+ 340us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.415722 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:51.418243 (+ 2521us) log.cc:842] Serialized 52 byte log entry
0806 20:15:51.642887 (+224644us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.642929 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.643001 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:52.480080 (+837079us) raft_consensus.cc:1671] finished
0806 20:15:52.480099 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:52.480235 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":952,"raft.queue_time_us":748,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":1748,"thread_start_us":194015,"threads_started":1,"wal-append.queue_time_us":441460}
I20260806 20:15:52.511374 30708 rpcz_store.cc:276] 0806 20:15:51.115222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.502377 (+1387155us) service_pool.cc:227] Handling call
0806 20:15:52.502634 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.509505 (+ 6871us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.509702 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.510509 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.482450 32574 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:52.485613 30864 raft_consensus.cc:2379] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:52.327824 32667 consensus_queue.cc:260] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.395794 32755 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:52.433529 32757 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:52.462436 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.452407 32672 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:52.486654 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.443475 30861 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:52.496909 30866 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:52.501070 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.526718 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.450023 32370 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:52.538142 336 log.cc:924] Time spent T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20260806 20:15:52.522522 31414 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
W20260806 20:15:52.585937 32610 consensus_meta.cc:287] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.504s user 0.006s sys 0.004s
I20260806 20:15:52.541702 30783 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:52.386372 32673 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.545591 32652 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:52.546455 32443 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:52.547266 32632 consensus_meta.cc:287] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.933s user 0.006s sys 0.004s
I20260806 20:15:52.547752 32507 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:52.552814 32668 consensus_meta.cc:287] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.926s user 0.007s sys 0.000s
I20260806 20:15:52.584861 30706 raft_consensus.cc:1261] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:15:52.738399 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1184) took 1546 ms. Trace:
I20260806 20:15:52.738780 30706 rpcz_store.cc:276] 0806 20:15:51.191755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.507938 (+1316183us) service_pool.cc:227] Handling call
0806 20:15:52.584475 (+ 76537us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.738086 (+153611us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.738338 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:52.437395 32767 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:52.742837 30789 raft_consensus.cc:2379] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:52.587783 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:52.594789 30603 catalog_manager.cc:5495] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 1 to 2, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 2 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:15:52.683027 32370 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.749363 30603 catalog_manager.cc:5495] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 2 to 3, leader changed from 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 3 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:52.751875 32370 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.612066 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.622416 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.752821 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.753557 30863 raft_consensus.cc:2379] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:52.683406 32667 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:52.754516 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.755084 32613 raft_consensus.cc:3032] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:52.755198 30863 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
W20260806 20:15:52.625905 334 log.cc:924] Time spent T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.402s user 0.000s sys 0.003s
I20260806 20:15:52.755586 32613 raft_consensus.cc:729] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:52.636953 30602 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41564 (request call id 227) took 9038 ms. Trace:
I20260806 20:15:52.756549 32613 consensus_queue.cc:260] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.641052 32580 raft_consensus.cc:3032] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:52.757611 32580 raft_consensus.cc:729] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:52.758697 32652 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:52.758473 32580 consensus_queue.cc:260] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.759646 32580 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:52.643787 32664 raft_consensus.cc:3032] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:15:52.600570 32717 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.652660 30871 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 20:15:52.760238 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1157) took 4203 ms. Trace:
I20260806 20:15:52.760667 30710 rpcz_store.cc:276] 0806 20:15:48.556845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.405783 (+2848938us) service_pool.cc:227] Handling call
0806 20:15:51.406074 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:51.711001 (+304927us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.759973 (+1048972us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54001b6054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:52.760160 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":554843392}
W20260806 20:15:52.761368 32708 consensus_meta.cc:287] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.792s user 0.008s sys 0.000s
I20260806 20:15:52.761962 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.763082 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.763520 32673 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:52.656977 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.538064 30713 raft_consensus.cc:2932] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } } }
I20260806 20:15:52.764737 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:52.537890 329 log.cc:924] Time spent T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.418s user 0.000s sys 0.001s
I20260806 20:15:52.673466 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.766352 30866 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
W20260806 20:15:52.766867 32274 consensus_meta.cc:287] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.857s user 0.009s sys 0.001s
I20260806 20:15:52.767227 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.768394 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.672559 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.769078 30866 raft_consensus.cc:2379] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:52.702313 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.770195 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.770533 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.771189 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.771637 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.772465 32613 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:52.772655 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.711688 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:52.652880 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.773711 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:52.773874 30864 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:52.774444 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1163) took 4215 ms. Trace:
I20260806 20:15:52.774746 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.774930 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:52.699285 32574 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:52.458549 32758 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.775460 30864 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:52.774839 32664 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:15:52.727797 32568 consensus_meta.cc:287] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.658s user 0.009s sys 0.000s
I20260806 20:15:52.776410 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.653513 32578 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:52.457356 32426 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.776724 32664 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.778062 32664 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:52.776098 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.778620 32426 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:52.738806 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:52.779460 30785 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:52.779301 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:52.780222 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.780323 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.780862 30789 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:52.738422 30869 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:52.705219 32640 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:52.781991 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:52.759864 32530 consensus_meta.cc:287] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.815s user 0.009s sys 0.000s
I20260806 20:15:52.653741 32755 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.784350 32755 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.756755 30602 rpcz_store.cc:276] 0806 20:15:43.598760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:43.598943 (+ 183us) service_pool.cc:227] Handling call
0806 20:15:52.636845 (+9037902us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:15:48.376226 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:15:48.376777 (+ 551us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:15:48.376814 (+ 37us) write_op.cc:435] Acquired schema lock
0806 20:15:48.388795 (+ 11981us) tablet.cc:660] Decoding operations
0806 20:15:48.431079 (+ 42284us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:15:48.431138 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0806 20:15:48.431167 (+ 29us) tablet.cc:683] Acquiring locks for 7 operations
0806 20:15:48.431739 (+ 572us) tablet.cc:699] Row locks acquired
0806 20:15:48.431758 (+ 19us) write_op.cc:260] PREPARE: finished
0806 20:15:48.431900 (+ 142us) write_op.cc:270] Start()
0806 20:15:48.432016 (+ 116us) write_op.cc:276] Timestamp: P: 1786047348431859 usec, L: 0
0806 20:15:48.432036 (+ 20us) op_driver.cc:331] REPLICATION: starting
0806 20:15:48.432685 (+ 649us) log.cc:842] Serialized 2891 byte log entry
0806 20:15:49.452524 (+1019839us) op_driver.cc:444] REPLICATION: finished
0806 20:15:49.876901 (+424377us) write_op.cc:301] APPLY: starting
0806 20:15:49.880003 (+ 3102us) tablet.cc:1365] starting BulkCheckPresence
0806 20:15:49.925229 (+ 45226us) tablet.cc:1368] finished BulkCheckPresence
0806 20:15:49.970046 (+ 44817us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:15:50.253209 (+283163us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:15:50.253391 (+ 182us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0806 20:15:50.253436 (+ 45us) write_op.cc:312] APPLY: finished
0806 20:15:50.254602 (+ 1166us) log.cc:842] Serialized 59 byte log entry
0806 20:15:50.254769 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:50.254882 (+ 113us) write_op.cc:454] Released schema lock
0806 20:15:50.254927 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0806 20:15:50.255190 (+ 263us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:50.255216 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":28710,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":385918,"apply.run_cpu_time_us":4374,"apply.run_wall_time_us":399692,"num_ops":7,"prepare.queue_time_us":211780,"prepare.run_cpu_time_us":42857,"prepare.run_wall_time_us":243024,"raft.queue_time_us":507098,"raft.run_cpu_time_us":14939,"raft.run_wall_time_us":274444,"replication_time_us":1008932,"thread_start_us":903132,"threads_started":4,"wal-append.queue_time_us":250020,"wal-append.run_cpu_time_us":38393,"wal-append.run_wall_time_us":2576494}]]}
I20260806 20:15:52.700815 32610 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:52.782269 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1320) took 1345 ms (client timeout 1717 ms). Trace:
I20260806 20:15:52.754504 32638 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:52.785846 30869 rpcz_store.cc:269] 0806 20:15:51.436432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.436580 (+ 148us) service_pool.cc:227] Handling call
0806 20:15:52.782106 (+1345526us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540019e454. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:52.782214 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2707074944}
I20260806 20:15:52.786955 32610 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:52.787060 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.778246 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.787694 32638 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.788087 30790 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:52.788352 30712 raft_consensus.cc:1226] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:52.767257 32274 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.782761 32530 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.790905 32530 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:52.791189 32274 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.792030 30784 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:52.792595 30791 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:52.794528 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1171) took 3091 ms. Trace:
I20260806 20:15:52.785243 30861 raft_consensus.cc:2396] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:15:52.761775 32708 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.794889 30712 rpcz_store.cc:276] 0806 20:15:49.703139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.489508 (+2786369us) service_pool.cc:227] Handling call
0806 20:15:52.489852 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.791398 (+301546us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.794334 (+ 2936us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54000c6a54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:52.794473 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":686094720}
I20260806 20:15:52.796386 32708 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:52.781286 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.797720 30789 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:52.798264 30785 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:52.798135 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.799000 30789 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:52.752358 30603 catalog_manager.cc:5495] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:52.801632 30603 catalog_manager.cc:5495] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 2 to 4, leader changed from 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 4 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:52.797804 30792 raft_consensus.cc:2379] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:52.775841 30711 rpcz_store.cc:276] 0806 20:15:48.559239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.554789 (+2995550us) service_pool.cc:227] Handling call
0806 20:15:51.555204 (+ 415us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:51.603872 (+ 48668us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.603908 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:51.604377 (+ 469us) log.cc:842] Serialized 26 byte log entry
0806 20:15:51.721756 (+117379us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.721835 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:51.721978 (+ 143us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:52.625241 (+903263us) spinlock_profiling.cc:243] Waited 6.18 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:15:52.774150 (+148909us) raft_consensus.cc:1671] finished
0806 20:15:52.774171 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:52.774383 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65845,"raft.queue_time_us":48348,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":13408,"spinlock_wait_cycles":14224256,"thread_start_us":71977,"threads_started":1,"wal-append.queue_time_us":399879}
I20260806 20:15:52.811421 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.812218 30787 raft_consensus.cc:2379] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:52.813211 30870 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:52.794543 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1162) took 4235 ms. Trace:
I20260806 20:15:52.817689 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.818923 30861 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:52.818037 30705 rpcz_store.cc:276] 0806 20:15:48.558952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.451455 (+2892503us) service_pool.cc:227] Handling call
0806 20:15:51.451840 (+ 385us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:51.642216 (+190376us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:51.645217 (+ 3001us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:51.650274 (+ 5057us) log.cc:842] Serialized 52 byte log entry
0806 20:15:51.905497 (+255223us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:51.915541 (+ 10044us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.643331 (+727790us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b540039eed4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:52.643382 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:52.794230 (+150848us) raft_consensus.cc:1671] finished
0806 20:15:52.794250 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:52.794474 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26774,"raft.queue_time_us":12483,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":8149,"spinlock_wait_cycles":283750144,"thread_start_us":202659,"threads_started":1,"wal-append.queue_time_us":682241}
W20260806 20:15:52.774703 32680 consensus_meta.cc:287] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.888s user 0.007s sys 0.004s
I20260806 20:15:52.820382 30861 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:52.820254 32680 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.821462 30791 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:52.822775 32680 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:52.790865 32758 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:52.823519 30791 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 20:15:52.824283 30787 raft_consensus.cc:1261] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 20:15:52.817335 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.826650 30870 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:52.829950 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.798959 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:52.810745 32574 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:52.741379 32767 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.836597 30870 raft_consensus.cc:1261] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:15:52.839722 30866 pending_rounds.cc:76] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 1
I20260806 20:15:52.842911 30866 pending_rounds.cc:97] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:15:52.835775 30861 raft_consensus.cc:1261] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:15:52.846218 30861 raft_consensus.cc:1226] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:52.848827 32671 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:52.849642 32671 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.851725 32671 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:52.852046 318 consensus_meta.cc:287] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.932s user 0.004s sys 0.005s
I20260806 20:15:52.852559 318 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.853891 30785 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:52.848774 30869 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:52.843300 30866 raft_consensus.cc:2866] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:52.856077 30869 raft_consensus.cc:1261] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260806 20:15:52.856655 30869 pending_rounds.cc:76] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 2
I20260806 20:15:52.856937 30869 pending_rounds.cc:97] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260806 20:15:52.857244 30869 raft_consensus.cc:2866] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:52.860752 30871 raft_consensus.cc:2396] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:52.853966 30786 raft_consensus.cc:2379] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:52.861833 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1280) took 1018 ms. Trace:
I20260806 20:15:52.862155 30786 rpcz_store.cc:276] 0806 20:15:51.843979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.846090 (+ 2111us) service_pool.cc:227] Handling call
0806 20:15:52.798798 (+952708us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:15:52.861774 (+ 62976us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10864512}
I20260806 20:15:52.865397 318 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.749473 32668 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.842499 30784 raft_consensus.cc:1226] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:52.879984 30868 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:52.885560 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1149) took 4763 ms. Trace:
I20260806 20:15:52.885959 30868 rpcz_store.cc:276] 0806 20:15:48.121958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:49.332594 (+1210636us) service_pool.cc:227] Handling call
0806 20:15:51.432819 (+2100225us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.880471 (+1447652us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.884744 (+ 4273us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b540007dd54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:52.885321 (+ 577us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b540007dd50. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:52.885487 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8157214464}
I20260806 20:15:52.887710 30868 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:52.889091 30868 raft_consensus.cc:1261] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 20:15:52.889487 30868 pending_rounds.cc:76] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 2
I20260806 20:15:52.889753 30868 pending_rounds.cc:97] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 20:15:52.890026 30868 raft_consensus.cc:2866] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:52.890353 30869 raft_consensus.cc:1261] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260806 20:15:52.890832 30869 pending_rounds.cc:76] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 2
I20260806 20:15:52.891047 30869 pending_rounds.cc:97] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260806 20:15:52.891064 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.874058 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.894606 30783 raft_consensus.cc:2379] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
W20260806 20:15:52.901592 30707 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.009s sys 0.017s
I20260806 20:15:52.835223 32757 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.892670 32647 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:52.891335 30869 raft_consensus.cc:2866] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:15:52.900274 30867 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.005s sys 0.022s
W20260806 20:15:52.922374 30867 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s user 0.005s sys 0.022s
W20260806 20:15:52.911782 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 574919us
W20260806 20:15:52.903443 30707 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s user 0.010s sys 0.017s
I20260806 20:15:52.923326 30709 raft_consensus.cc:1226] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:52.905416 32671 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:52.925819 30756 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:52.926335 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 155) took 1663 ms. Trace:
I20260806 20:15:52.926750 30709 rpcz_store.cc:276] 0806 20:15:51.263148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.512666 (+1249518us) service_pool.cc:227] Handling call
0806 20:15:52.654103 (+141437us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:52.923783 (+269680us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.926126 (+ 2343us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b5400175a54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:52.926281 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":618719104}
I20260806 20:15:52.927290 30756 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:52.928812 30789 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:52.930653 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 121) took 1039 ms. Trace:
I20260806 20:15:52.931329 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.933080 30869 raft_consensus.cc:2362] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4. Re-sending same reply.
I20260806 20:15:52.914055 32638 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:52.931023 30788 rpcz_store.cc:276] 0806 20:15:51.891132 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:51.891262 (+ 130us) service_pool.cc:227] Handling call
0806 20:15:52.109409 (+218147us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b5800004598. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:52.109616 (+ 207us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:52.187830 (+ 78214us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:52.187863 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:52.218489 (+ 30626us) log.cc:842] Serialized 26 byte log entry
0806 20:15:52.670025 (+451536us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:52.670071 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.670153 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:52.930322 (+260169us) raft_consensus.cc:1671] finished
0806 20:15:52.930345 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:52.930582 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6837,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":34538,"spinlock_wait_cycles":31212800,"thread_start_us":371213,"threads_started":1,"wal-append.queue_time_us":693801}
I20260806 20:15:52.953786 30756 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:52.956067 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 447650us
W20260806 20:15:52.957530 31388 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.016s sys 0.015s
I20260806 20:15:52.959062 30869 raft_consensus.cc:1226] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 20:15:52.931061 32767 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:52.933706 32755 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:52.933410 32758 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:52.974387 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:52.955183 32642 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:52.998739 30868 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
W20260806 20:15:52.961606 31388 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.506s user 0.016s sys 0.015s
W20260806 20:15:53.003834 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 548278us
W20260806 20:15:52.959450 339 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.025s sys 0.014s
I20260806 20:15:52.984002 30869 raft_consensus.cc:1226] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260806 20:15:52.933939 32671 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.963629 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:52.967525 31389 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:53.012804 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 56903us
I20260806 20:15:52.958706 32680 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20260806 20:15:52.955789 32638 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:52.973302 32758 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:53.018985 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:53.022112 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.065676 30709 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:15:53.019709 339 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.571s user 0.025s sys 0.014s
W20260806 20:15:53.032102 30754 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.003s sys 0.024s
W20260806 20:15:53.066843 30754 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.623s user 0.004s sys 0.024s
I20260806 20:15:53.001929 32647 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:53.064987 30869 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:53.072764 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:53.073977 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:53.014014 32668 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:52.979457 32755 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:53.067154 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 623048us
I20260806 20:15:53.078119 32568 consensus_queue.cc:1035] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:53.016872 32642 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260806 20:15:53.067607 32758 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:53.096915 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 374193us
I20260806 20:15:53.098486 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:53.099426 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:53.102281 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.103332 30708 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:53.079111 344 log.cc:924] Time spent T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260806 20:15:53.104176 30869 raft_consensus.cc:2362] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4. Re-sending same reply.
I20260806 20:15:53.107622 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:53.118953 30869 raft_consensus.cc:2379] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:53.110634 31422 raft_consensus.cc:2379] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:53.111739 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:53.120211 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:53.120620 30867 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:53.111397 30676 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.078487 32511 consensus_queue.cc:1046] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260806 20:15:53.066413 30863 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:53.117434 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:53.118868 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:53.068693 32274 consensus_queue.cc:1035] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260806 20:15:53.132038 30676 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.131474 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.121331 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:53.122051 30705 raft_consensus.cc:1261] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:15:53.133169 30710 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:53.123852 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.125976 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:53.127480 30787 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:53.135592 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:53.122987 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 378217us
I20260806 20:15:53.136325 30787 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:53.137010 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:15:53.137576 30787 raft_consensus.cc:595] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:15:53.108913 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.139240 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.140085 30711 raft_consensus.cc:2396] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:53.140821 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.115768 32238 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:53.141645 32717 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:53.148608 32238 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.149322 30676 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.151124 30785 raft_consensus.cc:1261] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:15:53.101905 32757 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:53.161013 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:53.166566 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.129982 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:53.131083 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:53.171880 30788 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:15:53.131253 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 319805us
I20260806 20:15:53.174847 30835 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.179839 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.123974 316 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:53.181110 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.185633 30705 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:53.183429 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.124133 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:53.188437 30869 raft_consensus.cc:2379] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:53.111346 32574 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.153635 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:53.178532 32647 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:53.161748 314 raft_consensus.cc:3032] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:53.177970 32733 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:53.202240 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:53.243331 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 75791.6us
I20260806 20:15:53.256259 30754 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:53.243813 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 75808.2us
I20260806 20:15:53.259127 30790 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:53.185385 32757 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:53.254988 30869 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:53.263489 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:53.264730 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:53.188122 32445 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:53.265467 30869 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:53.180075 32639 consensus_queue.cc:1035] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.077s
I20260806 20:15:53.195900 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:53.250689 31416 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:53.209131 32664 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:53.253978 32672 consensus_meta.cc:287] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.574s user 0.010s sys 0.000s
I20260806 20:15:53.206120 32511 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:53.253587 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:53.290334 30708 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
W20260806 20:15:53.296530 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 329060us
I20260806 20:15:53.332614 30709 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:53.332978 31389 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:53.300520 30867 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:53.303129 30705 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:53.244266 314 raft_consensus.cc:729] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:15:53.311733 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 162489us
I20260806 20:15:53.275764 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:53.190917 316 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.334437 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:53.266135 32733 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:53.333756 30710 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:53.354352 30676 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:53.355155 30867 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:53.274763 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:53.359254 31422 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:53.363384 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:53.355513 30869 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:53.364846 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.336581 30864 raft_consensus.cc:2396] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260806 20:15:53.417757 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 53013us
I20260806 20:15:53.420833 30867 raft_consensus.cc:1226] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
W20260806 20:15:53.386329 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 53411.8us
I20260806 20:15:53.423796 30676 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.340291 318 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:53.355707 32671 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:53.433230 30757 leader_election.cc:334] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.715s (SENT)
I20260806 20:15:53.412909 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:53.294610 32445 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.410797 337 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:53.361706 32638 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.517341 30867 raft_consensus.cc:1261] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 20:15:53.518410 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:53.510329 30869 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:53.421927 372 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:53.328220 32664 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:53.428597 353 log.cc:924] Time spent T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.236s user 0.001s sys 0.001s
I20260806 20:15:53.327656 32672 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.549619 32507 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:53.425405 32639 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:53.550297 32639 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:53.550180 32507 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.550781 32639 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:53.339766 32511 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:53.546965 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 372178us
I20260806 20:15:53.554672 30835 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:53.547607 362 log.cc:924] Time spent T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
W20260806 20:15:53.548754 361 log.cc:924] Time spent T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260806 20:15:53.549089 32750 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:15:53.549636 373 log.cc:924] Time spent T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
W20260806 20:15:53.562237 357 log.cc:924] Time spent T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.300s user 0.000s sys 0.001s
I20260806 20:15:53.417511 341 consensus_queue.cc:1035] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.104s
I20260806 20:15:53.564116 32511 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.416692 32733 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.565665 316 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:53.567188 32733 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.512847 337 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:53.567848 337 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:53.539166 314 consensus_queue.cc:260] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.572721 32672 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:53.573637 30703 raft_consensus.cc:1226] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
W20260806 20:15:53.577791 349 log.cc:924] Time spent T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.435s user 0.000s sys 0.001s
I20260806 20:15:53.558705 32664 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.584299 314 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:53.585843 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:53.586123 372 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.590283 30709 raft_consensus.cc:1226] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3. Ops: []
W20260806 20:15:53.591063 32580 consensus_meta.cc:287] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.795s user 0.007s sys 0.004s
I20260806 20:15:53.591337 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.539568 355 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:53.592440 355 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:53.540742 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:53.438778 32238 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.594080 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.595248 30865 raft_consensus.cc:2396] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:53.595723 30835 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:53.560276 32445 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.592617 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.633195 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1166) took 5027 ms. Trace:
I20260806 20:15:53.633491 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.637276 30711 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:53.638355 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.639473 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.640197 30711 raft_consensus.cc:2379] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:53.636992 30707 rpcz_store.cc:276] 0806 20:15:48.606265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.315609 (+3709344us) service_pool.cc:227] Handling call
0806 20:15:52.316009 (+ 400us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:52.316390 (+ 381us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:52.353259 (+ 36869us) log.cc:842] Serialized 14 byte log entry
0806 20:15:52.618469 (+265210us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b180019f0c0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e753764 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d 00007f71579ab204 00007f71579ab039 00007f714e735dc6
0806 20:15:52.923571 (+305102us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:52.924126 (+ 555us) log.cc:842] Serialized 52 byte log entry
0806 20:15:52.924649 (+ 523us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:52.924681 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:52.924749 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:53.593575 (+668826us) raft_consensus.cc:1671] finished
0806 20:15:53.632877 (+ 39302us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:53.633089 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":39020,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":994,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":13004,"spinlock_wait_cycles":6172032,"thread_start_us":534415,"threads_started":1,"wal-append.queue_time_us":720727}
I20260806 20:15:53.642446 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.642681 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.643242 30707 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:53.643316 30710 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:53.644182 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.644778 30707 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:53.647683 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.652966 30710 raft_consensus.cc:2396] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 20:15:53.655043 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.656916 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:53.659796 30757 leader_election.cc:334] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.749s (SENT)
I20260806 20:15:53.660470 30757 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:53.552527 32507 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:53.592815 355 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:53.659380 30707 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:53.662333 30710 raft_consensus.cc:2379] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 4: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:53.663426 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.664533 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.665382 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.553825 32755 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.666946 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.581337 32647 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 20:15:53.669130 30757 leader_election.cc:334] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.141s (SENT)
I20260806 20:15:53.614872 32580 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:15:53.670004 30757 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.196s (SENT) [suppressed 24 similar messages]
I20260806 20:15:53.670229 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.671029 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.672327 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.589141 372 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.675524 31421 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:53.675788 30711 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:15:53.675863 30757 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:53.678850 30710 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260806 20:15:53.643081 32750 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:53.696619 32750 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:53.697082 32750 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:53.697221 350 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:53.697676 32756 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:53.652908 316 consensus_queue.cc:1035] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260806 20:15:53.711910 32741 consensus_queue.cc:1046] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:15:53.713433 32580 raft_consensus.cc:3032] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:53.686715 318 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:53.652668 32618 consensus_queue.cc:1046] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260806 20:15:53.642185 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:53.682780 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1185) took 2490 ms. Trace:
I20260806 20:15:53.697947 350 consensus_queue.cc:1046] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:53.698604 32640 consensus_queue.cc:1046] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 20:15:53.700884 387 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:53.701612 385 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 20:15:53.702615 32758 consensus_meta.cc:287] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.587s user 0.010s sys 0.000s
I20260806 20:15:53.703214 32680 consensus_queue.cc:1046] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:53.701643 32642 consensus_queue.cc:1046] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 20:15:53.720506 350 consensus_queue.cc:1225] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer 58d59beb597f4ed99d84d102ddb41ff5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:15:53.720247 32758 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.722419 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:53.705368 30757 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:53.704816 32750 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.698729 32511 consensus_queue.cc:1035] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:53.716655 30835 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.718142 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:53.718024 30713 rpcz_store.cc:276] 0806 20:15:51.192058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.511906 (+1319848us) service_pool.cc:227] Handling call
0806 20:15:52.512263 (+ 357us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:52.537893 (+ 25630us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:52.796918 (+259025us) log.cc:842] Serialized 14 byte log entry
0806 20:15:53.150490 (+353572us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:53.178812 (+ 28322us) log.cc:842] Serialized 52 byte log entry
0806 20:15:53.178954 (+ 142us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:53.178986 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:53.179051 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:53.682550 (+503499us) raft_consensus.cc:1671] finished
0806 20:15:53.682567 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:53.682713 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11945,"raft.queue_time_us":21383,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":75681,"thread_start_us":334520,"threads_started":1,"wal-append.queue_time_us":460688}
I20260806 20:15:53.726343 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1332) took 1390 ms. Trace:
W20260806 20:15:53.727558 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 303841us
I20260806 20:15:53.726372 30835 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:53.728655 30676 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.726759 30866 rpcz_store.cc:276] 0806 20:15:52.336130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.830898 (+494768us) service_pool.cc:227] Handling call
0806 20:15:52.839534 (+ 8636us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:52.855433 (+ 15899us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:52.855463 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:52.856524 (+ 1061us) log.cc:842] Serialized 26 byte log entry
0806 20:15:53.193103 (+336579us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:53.193138 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:53.193195 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:53.726134 (+532939us) raft_consensus.cc:1671] finished
0806 20:15:53.726157 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:53.726281 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":314024,"threads_started":1,"wal-append.queue_time_us":827266}
I20260806 20:15:53.713868 32580 raft_consensus.cc:729] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:53.716037 32741 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:53.731933 30835 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:53.724828 32274 raft_consensus.cc:3032] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:15:53.732256 346 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:53.725781 32507 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:53.737222 346 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:53.737784 387 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:53.740746 32510 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:53.740918 346 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:53.741262 32750 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:53.717947 32639 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:53.742705 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 371747us
I20260806 20:15:53.731933 32758 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:53.743808 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 485649us
I20260806 20:15:53.734555 32673 raft_consensus.cc:3032] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:53.745631 30834 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.735288 32668 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:53.734108 32274 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:53.748212 32756 raft_consensus.cc:3032] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:15:53.748667 352 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:15:53.748149 30756 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:53.734704 32741 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.757326 30756 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:53.758288 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 127) took 1841 ms. Trace:
I20260806 20:15:53.760340 30790 rpcz_store.cc:276] 0806 20:15:51.916892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.788725 (+871833us) service_pool.cc:227] Handling call
0806 20:15:52.789007 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:53.259584 (+470577us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:53.758032 (+498448us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54001a2054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:53.758217 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1080722816}
W20260806 20:15:53.761899 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 338539us
I20260806 20:15:53.758867 32511 consensus_queue.cc:1035] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:15:53.764499 31388 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:53.766055 30756 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:53.764096 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 507887us
I20260806 20:15:53.758764 32758 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:53.740796 32580 consensus_queue.cc:260] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.753707 32668 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:53.753397 387 raft_consensus.cc:686] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:53.746831 32673 raft_consensus.cc:729] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:53.767807 30754 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:53.749356 32639 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:53.754737 32756 raft_consensus.cc:729] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:53.770859 30756 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:53.745724 32510 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.785014 372 consensus_queue.cc:1046] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260806 20:15:53.775511 32668 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:53.794359 352 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:15:53.799376 369 log.cc:924] Time spent T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.377s user 0.000s sys 0.003s
I20260806 20:15:53.768093 32680 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:53.801226 32580 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:53.813061 337 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.822512 401 raft_consensus.cc:3032] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:53.820698 32642 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:53.822991 401 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:53.823894 401 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.824038 355 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.825109 401 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:53.770474 385 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:53.786254 390 consensus_queue.cc:1046] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260806 20:15:53.826226 32741 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:53.815105 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 44321.3us
I20260806 20:15:53.826922 372 raft_consensus.cc:3032] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:53.827299 372 raft_consensus.cc:729] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:53.828055 372 consensus_queue.cc:260] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.830399 385 raft_consensus.cc:686] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:53.833465 385 consensus_queue.cc:237] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.833742 387 consensus_queue.cc:237] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.776810 32274 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.837728 352 raft_consensus.cc:686] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:53.838666 352 consensus_queue.cc:237] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.839440 32510 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:53.839919 32638 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:53.840654 32638 consensus_queue.cc:1046] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:15:53.785264 339 raft_consensus.cc:1050] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: attempting to promote NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 to VOTER
I20260806 20:15:53.825729 32756 consensus_queue.cc:260] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.841744 32638 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:53.845356 32756 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:53.847376 379 raft_consensus.cc:3032] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:53.847831 379 raft_consensus.cc:729] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:53.850317 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1334) took 1495 ms. Trace:
I20260806 20:15:53.850713 30870 rpcz_store.cc:276] 0806 20:15:52.354550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.844408 (+489858us) service_pool.cc:227] Handling call
0806 20:15:52.886994 (+ 42586us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:52.887447 (+ 453us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:52.901596 (+ 14149us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:52.901634 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:52.902211 (+ 577us) log.cc:842] Serialized 52 byte log entry
0806 20:15:53.177387 (+275176us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:53.177431 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:53.177528 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:53.772306 (+594778us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:15:53.850055 (+ 77749us) raft_consensus.cc:1671] finished
0806 20:15:53.850075 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:53.850245 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3517,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":1517,"spinlock_wait_cycles":14233856,"thread_start_us":247616,"threads_started":1,"wal-append.queue_time_us":387378}
I20260806 20:15:53.854384 32274 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:53.854475 32445 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:53.855571 32445 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.822723 318 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:53.829262 372 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:53.861057 391 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:53.862015 318 raft_consensus.cc:686] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:15:53.863313 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 134746us
I20260806 20:15:53.863973 398 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:53.864075 30676 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:53.819902 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 52187.5us
I20260806 20:15:53.870251 30754 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:53.848636 379 consensus_queue.cc:260] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.873992 30676 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.773152 32673 consensus_queue.cc:260] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.877683 30676 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.880551 30676 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:53.884116 30711 raft_consensus.cc:1226] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:53.882563 32639 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:53.887251 313 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:53.902041 355 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.865906 318 consensus_queue.cc:237] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.896857 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:53.895028 32680 consensus_queue.cc:1159] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "42cabca48db7403b8523a3382a046043" responder_term: 4 status { last_received { term: 4 index: 4 } last_committed_idx: 2 last_received_current_leader { term: 4 index: 4 } }
I20260806 20:15:53.905371 32647 raft_consensus.cc:3032] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:53.907832 32647 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:53.908885 32445 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.908643 32647 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.891978 32717 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260806 20:15:53.862049 391 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:53.912464 391 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:53.888563 32755 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:53.912873 32647 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:53.913228 32755 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:53.913638 32755 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:53.883795 32664 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:53.876976 32671 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:53.922974 30676 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:53.924093 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 53953us
I20260806 20:15:53.924002 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1186) took 1192 ms. Trace:
I20260806 20:15:53.926142 30865 rpcz_store.cc:276] 0806 20:15:52.731354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.764501 (+ 33147us) service_pool.cc:227] Handling call
0806 20:15:53.920711 (+1156210us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b540007e254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:53.923928 (+ 3217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1908020480}
I20260806 20:15:53.927066 30676 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.928236 30754 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:53.929126 32767 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:53.929526 30754 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:53.930300 32767 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:53.931021 379 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:53.923681 32668 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:53.932962 32668 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:53.930686 32767 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:53.933527 32767 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:53.936131 30676 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.936959 30754 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:53.924127 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 159706us
I20260806 20:15:53.954859 30676 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.959074 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:53.959743 31421 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:53.960436 30676 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:53.936372 32673 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:15:53.967773 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 200104us
I20260806 20:15:53.959657 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:53.945132 378 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:53.944523 388 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:53.980918 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:53.988732 30754 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:53.990157 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:53.992652 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:53.998926 30790 raft_consensus.cc:2379] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:53.956300 313 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:53.907230 346 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:54.002360 31389 leader_election.cc:334] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.851s (SENT)
I20260806 20:15:54.003937 30790 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:15:54.007319 30790 raft_consensus.cc:595] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:15:54.007718 30790 raft_consensus.cc:603] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:53.982841 32639 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:15:54.009290 31389 connection.cc:232] RPC call timeout handler was delayed by -1.2225502340006642s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
I20260806 20:15:53.966717 371 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:54.014263 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:54.015077 31419 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:15:54.013460 31389 connection.cc:232] RPC call timeout handler was delayed by -1.0418012639856897s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
W20260806 20:15:54.015975 31389 connection.cc:232] RPC call timeout handler was delayed by -1.0317502339894418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
I20260806 20:15:54.012943 30754 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:54.016280 31389 connection.cc:232] RPC call timeout handler was delayed by -1.0067012639774475s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15560000000000002s for a response
I20260806 20:15:54.018008 32638 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:54.018733 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1181) took 1288 ms. Trace:
I20260806 20:15:54.019322 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:53.995555 32672 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
I20260806 20:15:53.992079 378 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:53.983600 377 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:54.028353 31389 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:53.996142 337 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:54.036566 31389 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:53.985790 390 consensus_queue.cc:1046] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260806 20:15:54.010334 387 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:53.995101 32717 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
I20260806 20:15:54.066022 30710 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:54.068087 31419 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:54.030567 378 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:54.057557 377 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:54.060092 32767 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.082626 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:54.080199 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:54.084396 30710 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
W20260806 20:15:54.053705 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 73853.4us
I20260806 20:15:54.062974 32574 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:54.064257 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 321669us
I20260806 20:15:54.029806 32445 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:54.068243 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1198) took 1153 ms. Trace:
I20260806 20:15:54.091118 30868 rpcz_store.cc:276] 0806 20:15:52.915068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.999678 (+ 84610us) service_pool.cc:227] Handling call
0806 20:15:53.000010 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:53.000216 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:15:53.193389 (+193173us) log.cc:842] Serialized 14 byte log entry
0806 20:15:53.790236 (+596847us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b18004561c0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f71512fe82c 00007f714e7359dc 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb 00007f71512f6063 00007f71512f657b
0806 20:15:54.067922 (+277686us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:54.067970 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:54.068024 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:54.068083 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:54.068186 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":277443,"prepare.queue_time_us":426851,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":264,"raft.queue_time_us":281355,"spinlock_wait_cycles":12828032,"thread_start_us":427900,"threads_started":2,"wal-append.queue_time_us":380317}
I20260806 20:15:54.030750 32756 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:54.073680 30754 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:54.078783 30705 raft_consensus.cc:1261] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:15:54.088263 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 151) took 3119 ms. Trace:
I20260806 20:15:54.097818 30703 rpcz_store.cc:276] 0806 20:15:50.968972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.494897 (+1525925us) service_pool.cc:227] Handling call
0806 20:15:52.495221 (+ 324us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:53.574069 (+1078848us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:54.088010 (+513941us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001b4754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:54.088195 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2479720832}
I20260806 20:15:54.061983 339 consensus_queue.cc:237] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.027458 371 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.048695 387 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:54.098933 30754 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:54.046329 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.011868 313 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.107858 390 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:15:54.154789 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 409178us
I20260806 20:15:54.121048 32370 consensus_queue.cc:1046] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:15:54.121240 30676 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.019160 30864 rpcz_store.cc:276] 0806 20:15:52.730158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.776154 (+ 45996us) service_pool.cc:227] Handling call
0806 20:15:54.018514 (+1242360us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54000843d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:54.018665 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":918194432}
I20260806 20:15:54.073091 352 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:54.163101 30676 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.164319 30676 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:54.144690 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.145515 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.145766 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.146523 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.146692 30787 raft_consensus.cc:1261] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:15:54.147194 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.166867 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.167547 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:54.168314 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 69481.2us
W20260806 20:15:54.151456 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 114973us
I20260806 20:15:54.119053 32638 consensus_queue.cc:1035] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:54.174389 30791 raft_consensus.cc:1261] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:54.045055 404 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.104s
I20260806 20:15:54.078032 32741 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:15:54.115433 32574 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:54.077559 32667 consensus_queue.cc:1046] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.067s
I20260806 20:15:54.157081 30834 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.122895 387 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:54.147893 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.141487 398 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:54.138221 32664 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:54.169545 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.138389 30783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:15:54.143783 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1336) took 1685 ms. Trace:
I20260806 20:15:54.210685 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.318576 30783 raft_consensus.cc:595] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:54.122238 32750 consensus_queue.cc:1046] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260806 20:15:54.153774 32507 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:54.319911 32672 raft_consensus.cc:3032] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:54.320358 32672 raft_consensus.cc:729] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:54.320053 32507 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.179596 30787 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:54.122619 32580 consensus_queue.cc:1035] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260806 20:15:54.322412 32580 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:54.158615 421 raft_consensus.cc:3032] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:15:54.156656 32668 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:54.213042 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.323235 421 raft_consensus.cc:729] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:54.139158 32680 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260806 20:15:54.090945 377 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.326321 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:54.326726 377 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:54.168525 32756 raft_consensus.cc:3032] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:54.327368 32756 raft_consensus.cc:729] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:15:54.222189 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 57938.5us
I20260806 20:15:54.175531 32717 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:54.215667 32370 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:54.165782 401 consensus_queue.cc:1046] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260806 20:15:54.226720 32741 consensus_queue.cc:1035] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:15:54.210873 318 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 20:15:54.228447 402 log.cc:924] Time spent T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Append to log took a long time: real 0.203s user 0.000s sys 0.001s
I20260806 20:15:54.335198 337 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:54.335729 337 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:54.229156 413 log.cc:924] Time spent T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
W20260806 20:15:54.229918 412 log.cc:924] Time spent T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 20:15:54.232769 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1211) took 1063 ms. Trace:
W20260806 20:15:54.235284 415 log.cc:924] Time spent T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20260806 20:15:54.236840 408 log.cc:924] Time spent T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260806 20:15:54.315956 352 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:54.246930 30792 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:54.263080 371 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:54.263639 32443 consensus_queue.cc:1035] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:54.264469 32638 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:54.211704 32510 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:54.211347 418 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:54.225230 32755 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.264326 30784 raft_consensus.cc:1226] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:54.274659 313 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:54.261746 346 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:54.299048 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 142892us
I20260806 20:15:54.359611 30834 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:54.312501 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 143046us
I20260806 20:15:54.360612 31389 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:54.360695 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 290449us
I20260806 20:15:54.328922 30676 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.362048 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.338014 30835 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:54.362839 31389 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.344708 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.365429 30787 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:54.368593 30835 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:54.368758 30834 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.341578 32574 raft_consensus.cc:686] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:15:54.346143 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 105646us
I20260806 20:15:54.390363 30834 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.238794 32767 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:54.300904 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 131794us
I20260806 20:15:54.397506 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:54.398391 31423 raft_consensus.cc:2379] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:54.321146 32672 consensus_queue.cc:260] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.376174 30787 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:54.344102 32370 raft_consensus.cc:686] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:54.387835 352 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:54.387991 32717 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:54.331770 32668 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.364809 31389 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:54.360776 32510 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.387264 371 raft_consensus.cc:2781] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:15:54.347363 387 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.400902 346 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:54.328249 32756 consensus_queue.cc:260] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.412359 30788 raft_consensus.cc:2396] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260806 20:15:54.416851 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 52125.1us
I20260806 20:15:54.427829 31389 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:54.435067 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:54.436076 31422 raft_consensus.cc:2396] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:15:54.411952 32639 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:54.423607 346 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:54.436826 31423 raft_consensus.cc:1226] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:54.409124 337 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:54.414036 32574 consensus_queue.cc:237] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.418339 32510 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:54.437114 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:54.419121 404 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:15:54.418746 378 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.440552 30832 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.442379 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:54.443500 31423 raft_consensus.cc:2379] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:54.448868 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:54.449807 31421 raft_consensus.cc:2379] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:54.408377 32667 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:54.424297 32750 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:54.419042 371 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:54.423218 387 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:54.360903 32638 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.436126 439 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:15:54.415627 32668 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:54.439476 390 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:54.409150 32507 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:54.457928 32756 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:54.412812 352 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:54.442849 346 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:54.437623 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:54.446990 32741 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:54.460677 371 raft_consensus.cc:3037] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:54.451576 32510 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:54.409673 32755 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:54.412766 32370 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.466091 32667 raft_consensus.cc:686] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:54.468997 390 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:54.471820 352 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:54.473061 378 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:54.442221 32647 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
I20260806 20:15:54.464073 32750 raft_consensus.cc:686] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:54.405992 424 consensus_queue.cc:1046] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260806 20:15:54.482982 390 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:54.491340 32667 consensus_queue.cc:237] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.507427 32750 consensus_queue.cc:237] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.525960 32638 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:54.542356 352 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.545748 352 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:54.547653 390 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.550360 390 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:54.552485 371 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.554360 418 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:54.343797 30708 rpcz_store.cc:276] 0806 20:15:53.169264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:53.335188 (+165924us) service_pool.cc:227] Handling call
0806 20:15:53.335499 (+ 311us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:53.849798 (+514299us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:53.849832 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:53.850467 (+ 635us) log.cc:842] Serialized 52 byte log entry
0806 20:15:53.991014 (+140547us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:15:53.991051 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:54.058705 (+ 67654us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b5400175554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:54.058756 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:54.232460 (+173704us) raft_consensus.cc:1671] finished
0806 20:15:54.232479 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:54.232706 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":293144,"raft.queue_time_us":294529,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":158,"spinlock_wait_cycles":507522816,"thread_start_us":140161,"threads_started":1,"wal-append.queue_time_us":315081}
I20260806 20:15:54.609021 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:54.422362 30786 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:54.615952 30787 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:54.621733 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.622339 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.623121 30787 raft_consensus.cc:2396] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:54.623519 30784 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
W20260806 20:15:54.612660 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 235292us
I20260806 20:15:54.629755 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:54.631322 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 240995us
I20260806 20:15:54.632745 30834 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.632867 30835 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:54.634662 30710 raft_consensus.cc:1261] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:54.635092 30834 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.637491 30834 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.642621 30864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f32336173df4611829be37d396e86ba"
dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47072
I20260806 20:15:54.643782 30864 raft_consensus.cc:595] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:15:54.318536 30861 rpcz_store.cc:276] 0806 20:15:52.454535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:52.846976 (+392441us) service_pool.cc:227] Handling call
0806 20:15:52.847339 (+ 363us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:52.847780 (+ 441us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:52.847816 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:52.848481 (+ 665us) log.cc:842] Serialized 52 byte log entry
0806 20:15:53.307252 (+458771us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:15:53.732359 (+425107us) log.cc:842] Serialized 14 byte log entry
0806 20:15:53.741623 (+ 9264us) log.cc:842] Serialized 14 byte log entry
0806 20:15:54.063678 (+322055us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b18002d75a0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f714e58e990 00007f714e750864 00007f714e750155 00007f71516673d0 00007f7151679576 00007f715168a0d7 00007f715168a058
0806 20:15:54.063911 (+ 233us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:54.063993 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:54.107059 (+ 43066us) raft_consensus.cc:1671] finished
0806 20:15:54.107089 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:54.139585 (+ 32496us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":578,"prepare.queue_time_us":228065,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":113,"raft.queue_time_us":327751,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":10215,"spinlock_wait_cycles":6154624,"thread_start_us":736326,"threads_started":2,"wal-append.queue_time_us":698837}
I20260806 20:15:54.644039 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.648408 30864 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a439c5debb41440c913a50da91e6467f"
dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47072
I20260806 20:15:54.649286 30864 raft_consensus.cc:595] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:15:54.649844 30864 raft_consensus.cc:603] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:54.650704 30835 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:54.651242 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:15:54.652515 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.653688 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.658130 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.661940 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.663086 30868 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:54.672539 30709 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:15:54.674062 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:54.682243 31422 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEARNER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 20:15:54.683359 32238 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:54.685961 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:54.686904 31423 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:54.706908 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.639971 30834 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.718524 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.719853 30861 raft_consensus.cc:2396] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 7 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:54.719693 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.720749 30870 raft_consensus.cc:2379] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:54.720960 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.721858 30866 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:54.710289 32507 consensus_queue.cc:1046] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260806 20:15:54.725842 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:54.726084 390 consensus_queue.cc:1046] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:15:54.726812 31414 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:54.728231 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:54.728571 390 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:15:54.729265 30708 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:54.729408 30707 raft_consensus.cc:2396] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 20:15:54.731941 32667 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:54.732290 424 raft_consensus.cc:3032] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:54.732806 424 raft_consensus.cc:729] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:54.733004 32667 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:54.733741 32667 raft_consensus.cc:686] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:54.738117 387 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:54.741113 30868 raft_consensus.cc:2379] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:54.728422 30787 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:15:54.723397 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.735280 32667 consensus_queue.cc:237] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.768431 32668 consensus_queue.cc:1046] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:54.769345 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.770027 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:54.770824 30868 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:54.780694 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.789094 30864 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:54.790272 439 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:54.790805 439 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:54.791136 439 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:54.784425 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.785703 30783 raft_consensus.cc:1226] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260806 20:15:54.793991 32741 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:54.794493 32741 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:54.794947 32741 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:54.797508 432 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.017s
I20260806 20:15:54.798506 32507 raft_consensus.cc:3032] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:54.798944 32507 raft_consensus.cc:729] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:54.800145 32507 consensus_queue.cc:260] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.801782 30710 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
W20260806 20:15:54.787156 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 61727.1us
I20260806 20:15:54.803444 371 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:54.804685 30673 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:54.806289 32647 raft_consensus.cc:3032] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:15:54.806711 32647 raft_consensus.cc:729] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:54.770040 31416 raft_consensus.cc:1226] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:54.818660 430 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:15:54.819972 432 consensus_queue.cc:1046] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260806 20:15:54.821440 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 181705us
I20260806 20:15:54.823948 31415 raft_consensus.cc:1226] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:54.768548 346 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.832851 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.830230 430 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:54.821199 32667 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:15:54.835738 31423 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:54.837468 430 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:54.784307 424 consensus_queue.cc:260] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.846781 30708 raft_consensus.cc:1261] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:15:54.848104 30834 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.849102 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.849975 30792 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:54.850462 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.851006 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:54.851475 30790 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:54.852769 30792 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:54.843825 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:54.868696 435 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:54.870335 30708 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 20:15:54.873404 30865 raft_consensus.cc:2379] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:54.867424 30710 raft_consensus.cc:1261] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:54.807983 32647 consensus_queue.cc:260] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:54.863646 32667 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:54.886066 32741 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:54.886096 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.903126 32757 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:54.903643 434 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:54.892894 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.892853 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:54.903350 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.935694 30754 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:54.982687 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 45904.8us
I20260806 20:15:54.987643 30868 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:54.985461 32757 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } attrs { replace: true } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:54.993984 30863 raft_consensus.cc:2396] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:54.995102 30834 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:54.995290 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:54.996088 30863 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:54.985741 435 consensus_queue.cc:1046] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260806 20:15:54.997982 441 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:54.998339 421 consensus_queue.cc:260] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:55.000124 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 81810.3us
I20260806 20:15:55.001885 434 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.003248 346 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.003868 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:55.003846 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 129378us
I20260806 20:15:55.014424 30866 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:54.996271 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:55.004379 32647 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.003898 32757 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.005816 441 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:55.026331 353 log.cc:924] Time spent T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.235s user 0.001s sys 0.003s
I20260806 20:15:55.015522 435 consensus_queue.cc:1225] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer d4181d4d907f477f9feddc44ba3b9207 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 20:15:55.036453 32443 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:55.037688 339 consensus_queue.cc:1046] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260806 20:15:55.038147 439 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.039083 339 raft_consensus.cc:3032] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:55.039434 339 raft_consensus.cc:729] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:55.040735 434 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:55.040576 339 consensus_queue.cc:260] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.041771 425 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:55.036480 32640 consensus_queue.cc:1035] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20260806 20:15:55.042533 425 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.044971 425 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.022141 32741 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:55.045945 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 48605.5us
I20260806 20:15:55.047020 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.046881 30784 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:55.025724 441 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:55.078145 30868 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:55.061873 439 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.081564 314 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:55.082146 314 raft_consensus.cc:686] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:55.083079 439 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:15:55.083565 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 156232us
I20260806 20:15:55.083379 314 consensus_queue.cc:237] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.085580 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:55.087047 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:55.088610 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:55.089144 30784 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:15:55.089922 456 log.cc:924] Time spent T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20260806 20:15:55.089993 32642 consensus_queue.cc:1046] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:15:55.090917 32238 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:55.091487 32642 raft_consensus.cc:3032] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:55.091919 32642 raft_consensus.cc:729] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:55.092466 30865 raft_consensus.cc:1226] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:55.045513 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:55.094213 339 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:55.099973 350 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:55.100093 32756 consensus_queue.cc:1046] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:55.108613 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:55.108498 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 131126us
I20260806 20:15:55.101059 385 consensus_queue.cc:1035] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20260806 20:15:55.118003 32640 consensus_queue.cc:1035] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:55.120575 32640 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.120965 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:55.121188 32640 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.125645 32640 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.136087 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:55.137487 30783 raft_consensus.cc:2396] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:15:55.138861 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:55.093009 32642 consensus_queue.cc:260] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:55.149745 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 45837.2us
I20260806 20:15:55.102666 32741 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:55.155381 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:55.159511 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:55.170017 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:55.173327 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 98634.5us
I20260806 20:15:55.174384 30834 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.152114 439 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.175894 30834 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:55.179088 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:15:55.179584 32750 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.180245 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:15:55.180454 439 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:55.181850 32642 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:55.183338 439 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:55.184000 439 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:15:55.173539 463 log.cc:924] Time spent T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260806 20:15:55.170670 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:55.201709 441 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.203919 441 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.213809 30676 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.218681 32580 raft_consensus.cc:3032] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:55.219245 32580 raft_consensus.cc:729] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:55.220317 32580 consensus_queue.cc:260] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.202128 30787 raft_consensus.cc:2379] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:55.180281 32750 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.198685 30709 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:55.235997 30756 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:55.187676 434 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:55.222913 31415 raft_consensus.cc:1226] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
I20260806 20:15:55.227916 32580 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:55.243546 32238 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:55.253254 32756 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.271067 318 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:55.271639 318 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:55.271992 318 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:15:55.276862 30757 leader_election.cc:334] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.620s (SENT)
I20260806 20:15:55.277487 30757 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:55.261795 434 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:55.304865 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 125849us
I20260806 20:15:55.325975 30676 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.327925 32638 consensus_queue.cc:1046] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:55.328922 32638 raft_consensus.cc:3032] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:55.329248 32638 raft_consensus.cc:729] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:55.330116 32638 consensus_queue.cc:260] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.333706 30784 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:55.334100 32638 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:55.327867 30676 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:55.353003 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 206975us
I20260806 20:15:55.306391 32756 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.310981 430 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.356609 32757 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:55.390570 32733 consensus_queue.cc:1046] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260806 20:15:55.391783 32733 raft_consensus.cc:3032] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:15:55.392172 32733 raft_consensus.cc:729] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:55.394932 430 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:55.398149 32750 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.372102 434 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.366768 425 raft_consensus.cc:2741] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 20:15:55.381484 30676 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.382893 31414 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
W20260806 20:15:55.384006 469 log.cc:924] Time spent T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260806 20:15:55.384410 388 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:15:55.385488 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 108675us
I20260806 20:15:55.404163 30835 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:55.397569 30786 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53732 (request call id 136) took 1401 ms (client timeout 1695 ms). Trace:
I20260806 20:15:55.418517 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:55.419682 30867 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:55.419751 32667 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:55.420630 32667 raft_consensus.cc:686] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:55.422031 32667 consensus_queue.cc:237] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.427781 30784 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:55.429780 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:55.430625 30785 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:55.431175 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:55.431370 30791 raft_consensus.cc:1261] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:15:55.430697 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:55.404542 31414 pending_rounds.cc:76] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Aborting all ops after (but not including) 6
W20260806 20:15:55.435472 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 259694us
I20260806 20:15:55.406702 434 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.400813 32756 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:55.448810 30757 leader_election.cc:334] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.657s (SENT)
I20260806 20:15:55.449347 30757 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:55.451200 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:55.443625 31414 pending_rounds.cc:97] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Aborting uncommitted NO_OP operation due to leader change: 4.7
W20260806 20:15:55.417290 30786 rpcz_store.cc:269] 0806 20:15:53.996092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:53.998087 (+ 1995us) service_pool.cc:227] Handling call
0806 20:15:55.397140 (+1399053us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b540026f054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:55.397339 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":623107328}
W20260806 20:15:55.457643 30788 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1613) took 1313 ms (client timeout 1659 ms). Trace:
W20260806 20:15:55.458079 30788 rpcz_store.cc:269] 0806 20:15:54.143975 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:54.146970 (+ 2995us) service_pool.cc:227] Handling call
0806 20:15:55.457158 (+1310188us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54001d1fd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:55.457424 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":562889600}
I20260806 20:15:55.460817 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:55.463660 404 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.464401 404 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.393117 32733 consensus_queue.cc:260] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.453871 30713 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:55.450284 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:55.473479 31421 raft_consensus.cc:1226] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:55.456815 31414 raft_consensus.cc:2866] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:55.441877 32668 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:55.485867 32668 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.488278 32668 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:55.489615 30832 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:55.490449 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 109101us
I20260806 20:15:55.474267 30784 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:55.476985 30756 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:55.474104 441 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.467672 434 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.474097 32750 consensus_queue.cc:1046] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260806 20:15:55.481529 435 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:15:55.483292 32756 raft_consensus.cc:2932] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:55.494252 31416 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:15:55.494805 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:55.499406 31423 raft_consensus.cc:2379] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:55.480513 314 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:55.479149 31415 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:55.496404 388 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:15:55.503299 30835 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.504606 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:55.505617 32668 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:55.502558 441 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.506419 32668 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:55.506821 32668 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:55.507148 32668 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.507798 441 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:55.497962 32642 consensus_queue.cc:1046] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260806 20:15:55.502660 32750 consensus_queue.cc:1225] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer d4181d4d907f477f9feddc44ba3b9207 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 20:15:55.510123 391 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:15:55.501844 346 consensus_queue.cc:1035] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.013s
I20260806 20:15:55.515534 30788 raft_consensus.cc:1261] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:55.515918 478 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:55.503031 388 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:55.475324 318 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.529778 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:55.509900 30708 raft_consensus.cc:2932] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:15:55.539795 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 91048.8us
W20260806 20:15:55.541324 30757 leader_election.cc:334] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.664s (SENT)
W20260806 20:15:55.542850 30757 leader_election.cc:334] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.732s (SENT)
I20260806 20:15:55.544715 32664 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.545399 32664 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.541271 426 raft_consensus.cc:3032] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:15:55.546370 426 raft_consensus.cc:729] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:55.547626 32664 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.549145 30834 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.551927 32672 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:55.552390 489 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:55.527976 346 consensus_queue.cc:1035] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260806 20:15:55.529548 32733 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:55.557821 479 consensus_queue.cc:1046] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:15:55.560475 391 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:15:55.561954 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:55.562721 30786 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:55.563839 318 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.564905 30603 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41550 (request call id 237) took 9490 ms. Trace:
I20260806 20:15:55.565102 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:55.571727 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 47072.3us
I20260806 20:15:55.572084 30703 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.565261 404 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.505654 434 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.529044 388 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:55.538897 387 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260806 20:15:55.537861 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:55.579387 31416 raft_consensus.cc:2379] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:55.574090 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:55.580219 30709 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:55.552006 30834 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.581396 31388 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:55.587667 30835 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:55.589161 30835 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.590956 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.592306 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.594337 430 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:55.594756 430 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:55.595108 430 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.597654 32664 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:15:55.599787 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.603140 425 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:55.603582 425 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:55.603955 425 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:55.611366 30676 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.614099 32664 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:55.622216 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.627801 32445 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:55.628198 32445 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:55.628566 32445 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:55.638873 32445 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.580283 32568 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:55.641197 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 52114.4us
I20260806 20:15:55.644375 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:55.645184 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 63386.4us
I20260806 20:15:55.645793 352 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.648248 32667 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:15:55.648514 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 67227us
I20260806 20:15:55.572681 32668 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.649644 32647 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:55.626724 32664 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:15:55.655938 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 69700.6us
I20260806 20:15:55.585373 404 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.680081 404 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.680874 434 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.648662 439 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.654004 346 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:55.687716 404 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.688589 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:55.689338 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:55.689584 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:55.689983 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:55.690356 30861 raft_consensus.cc:1261] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 20:15:55.690568 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:55.691231 30871 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:55.692250 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:55.681665 30787 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:55.693898 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:55.694303 318 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.694855 318 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.685247 401 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.696043 401 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.697049 31415 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:15:55.709725 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:55.711813 385 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.712414 385 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.697786 390 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.714991 30673 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:55.715483 390 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.719730 30870 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:55.720432 32638 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:15:55.700179 426 consensus_queue.cc:260] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.739065 30703 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:55.661697 30676 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.703271 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1213) took 1844 ms. Trace:
I20260806 20:15:55.705076 30861 pending_rounds.cc:76] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 2
I20260806 20:15:55.705639 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:55.571493 30603 rpcz_store.cc:276] 0806 20:15:46.074057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:46.074348 (+ 291us) service_pool.cc:227] Handling call
0806 20:15:55.564631 (+9490283us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:15:53.431208 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:15:53.575993 (+144785us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:15:53.576038 (+ 45us) write_op.cc:435] Acquired schema lock
0806 20:15:53.576051 (+ 13us) tablet.cc:660] Decoding operations
0806 20:15:53.576711 (+ 660us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:15:53.576768 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0806 20:15:53.576799 (+ 31us) tablet.cc:683] Acquiring locks for 4 operations
0806 20:15:53.577209 (+ 410us) tablet.cc:699] Row locks acquired
0806 20:15:53.577224 (+ 15us) write_op.cc:260] PREPARE: finished
0806 20:15:53.577385 (+ 161us) write_op.cc:270] Start()
0806 20:15:53.634139 (+ 56754us) write_op.cc:276] Timestamp: P: 1786047353577330 usec, L: 0
0806 20:15:53.634163 (+ 24us) op_driver.cc:331] REPLICATION: starting
0806 20:15:53.675362 (+ 41199us) log.cc:842] Serialized 1708 byte log entry
0806 20:15:54.490937 (+815575us) op_driver.cc:444] REPLICATION: finished
0806 20:15:54.556397 (+ 65460us) write_op.cc:301] APPLY: starting
0806 20:15:54.556496 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0806 20:15:54.556664 (+ 168us) tablet.cc:1368] finished BulkCheckPresence
0806 20:15:54.556810 (+ 146us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:15:54.557408 (+ 598us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:15:54.557597 (+ 189us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0806 20:15:54.557644 (+ 47us) write_op.cc:312] APPLY: finished
0806 20:15:54.557901 (+ 257us) log.cc:842] Serialized 41 byte log entry
0806 20:15:54.558049 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:54.558146 (+ 97us) write_op.cc:454] Released schema lock
0806 20:15:54.558195 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0806 20:15:54.796221 (+238026us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:54.796303 (+ 82us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1300123,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":65155,"apply.run_cpu_time_us":3194,"apply.run_wall_time_us":240471,"num_ops":4,"prepare.queue_time_us":504832,"prepare.run_cpu_time_us":22499,"prepare.run_wall_time_us":479780,"raft.queue_time_us":154383,"raft.run_cpu_time_us":6581,"raft.run_wall_time_us":65466,"replication_time_us":848543,"thread_start_us":765851,"threads_started":4,"wal-append.queue_time_us":345794}]]}
I20260806 20:15:55.656494 391 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.672082 32568 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.673872 32647 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:55.670667 352 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.661257 32667 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:55.698179 401 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.789139 32647 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:55.804908 30869 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:15:55.744231 30864 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:55.746135 31415 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 20:15:55.746745 457 log.cc:924] Time spent T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.730s user 0.003s sys 0.000s
I20260806 20:15:55.750897 30866 raft_consensus.cc:2354] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6.
I20260806 20:15:55.779534 30865 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:55.786402 391 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.787366 32668 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.788566 32568 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.743465 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:55.790468 352 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.720942 32638 raft_consensus.cc:686] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:55.826570 30870 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:55.752077 30706 rpcz_store.cc:276] 0806 20:15:53.859237 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:53.868418 (+ 9181us) service_pool.cc:227] Handling call
0806 20:15:53.882981 (+ 14563us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:53.920773 (+ 37792us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:15:53.944559 (+ 23786us) log.cc:842] Serialized 14 byte log entry
0806 20:15:54.074444 (+129885us) log.cc:842] Serialized 14 byte log entry
0806 20:15:54.075039 (+ 595us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:54.075167 (+ 128us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:54.075243 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:54.339051 (+263808us) spinlock_profiling.cc:243] Waited 6 ms on lock 0x7b180032a400. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e720f96 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9
0806 20:15:55.702793 (+1363742us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b54000c6f54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:55.702865 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:55.703060 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":116702,"prepare.queue_time_us":59742,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":7957,"raft.queue_time_us":714,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":2548,"spinlock_wait_cycles":100196864,"thread_start_us":88692,"threads_started":1,"wal-append.queue_time_us":382747,"wal-append.run_cpu_time_us":1847,"wal-append.run_wall_time_us":1048160}
I20260806 20:15:55.659895 430 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.802242 350 raft_consensus.cc:3032] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:55.791899 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:55.792562 30787 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:55.807413 30871 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:55.743813 385 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:55.815932 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:55.779819 30861 pending_rounds.cc:97] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 20:15:55.767522 439 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:15:55.818991 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 175612us
I20260806 20:15:55.801736 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:55.843158 30713 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:15:55.817714 421 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.848183 30676 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.836253 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:55.849745 30835 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:55.828353 30756 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:55.798080 32757 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:55.800287 318 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:55.812587 32667 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:55.812168 32672 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.835093 31423 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:55.781845 390 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:55.801244 328 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.854811 30709 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:55.867049 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.867275 30754 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:55.834784 350 raft_consensus.cc:729] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:55.842391 424 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:55.842536 30861 raft_consensus.cc:2866] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:55.876128 31388 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:55.878283 31388 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:55.880926 318 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.881311 31423 raft_consensus.cc:1226] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:55.881507 318 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.856961 32507 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:55.858566 30786 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.863299 32741 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:55.867067 32568 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260806 20:15:55.889657 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.895838 30676 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:55.889340 32741 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:55.895507 30602 catalog_manager.cc:5495] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 2 to 4, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 4 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:15:55.896481 32741 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.897658 30602 catalog_manager.cc:5495] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 5, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 5 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:55.899374 318 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.899575 30602 catalog_manager.cc:5495] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:15:55.864097 32672 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:55.901960 32672 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:55.902514 32672 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:55.902878 32672 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:55.903267 32672 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.882628 32640 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:55.882678 30863 raft_consensus.cc:1226] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260806 20:15:55.919519 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:55.885576 339 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:55.925776 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:55.928263 30713 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:55.928953 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:55.891724 426 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:55.893572 390 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:55.887293 32757 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.932927 30602 catalog_manager.cc:5495] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:55.945235 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:55.917537 32445 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:55.920265 489 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:55.919754 32238 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:55.959331 30703 raft_consensus.cc:1226] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:55.955925 30708 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:55.959121 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:55.930817 32664 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:55.931841 339 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:55.985925 30710 raft_consensus.cc:1226] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:55.902132 425 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.990108 30706 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:55.954855 30864 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260806 20:15:55.993162 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:55.948938 32640 raft_consensus.cc:686] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:55.949905 390 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:56.005012 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:56.019799 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 124076us
I20260806 20:15:56.026167 30676 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:55.926527 388 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:55.994890 30602 catalog_manager.cc:5495] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 4 to 5, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 5 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.018217 30756 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:56.028107 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 170649us
I20260806 20:15:55.998098 339 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:56.030254 31419 raft_consensus.cc:2932] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:56.032016 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:55.933933 32638 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.028138 30709 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
W20260806 20:15:56.041189 30757 leader_election.cc:334] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.196s (SENT)
I20260806 20:15:56.044313 30757 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:56.027884 30602 catalog_manager.cc:5495] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 5, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 5 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.017477 32664 raft_consensus.cc:686] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:56.047101 30713 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:56.027704 32717 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:56.028391 390 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:56.053357 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.002372 32757 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.061501 30602 catalog_manager.cc:5495] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 3 to 4, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.078800 30602 catalog_manager.cc:5495] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.047295 442 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:56.050320 32642 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:56.082108 30756 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:56.086409 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.045383 430 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:56.093060 30602 catalog_manager.cc:5495] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 3, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 3 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.123435 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:56.092528 32568 consensus_queue.cc:1046] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
W20260806 20:15:56.153486 498 log.cc:924] Time spent T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.204s user 0.001s sys 0.000s
I20260806 20:15:56.080399 391 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:56.120698 30676 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.154358 32574 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:56.154785 32574 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:56.155102 32574 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:56.132750 30788 raft_consensus.cc:1261] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:56.015807 489 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:56.158501 495 log.cc:924] Time spent T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20260806 20:15:56.157891 30602 catalog_manager.cc:5495] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 4, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:55.875339 328 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.160053 391 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:56.159668 30602 catalog_manager.cc:5495] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 3 to 4, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.160583 391 raft_consensus.cc:686] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:56.161199 30789 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:56.161314 328 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:56.009912 350 consensus_queue.cc:260] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.058490 32445 consensus_queue.cc:1046] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.061s
W20260806 20:15:56.162798 501 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20260806 20:15:56.163049 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:56.164294 499 log.cc:924] Time spent T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.234s user 0.000s sys 0.001s
I20260806 20:15:56.165068 391 consensus_queue.cc:237] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.153721 32568 raft_consensus.cc:3032] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:56.167521 32568 raft_consensus.cc:729] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:56.168486 32568 consensus_queue.cc:260] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.169687 32568 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:56.172478 32445 raft_consensus.cc:3032] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:56.172883 32445 raft_consensus.cc:729] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:56.173817 32445 consensus_queue.cc:260] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.175092 32445 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:56.174312 30602 catalog_manager.cc:5495] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.176362 32672 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.178390 32757 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:56.142683 32640 consensus_queue.cc:237] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.095314 442 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:56.121371 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.092558 439 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.092630 32642 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:56.136579 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 102557us
I20260806 20:15:56.135669 32274 consensus_queue.cc:1046] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260806 20:15:56.145025 377 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:56.136739 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:56.146144 496 log.cc:924] Time spent T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20260806 20:15:56.146989 32667 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.115334 30785 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:56.176656 30602 catalog_manager.cc:5495] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
W20260806 20:15:56.153012 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 70994us
I20260806 20:15:56.162272 351 consensus_queue.cc:1046] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:56.182991 388 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:56.194058 32642 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:56.206220 30864 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:56.206979 32741 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.207288 30602 catalog_manager.cc:5495] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 2 to 3, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 3 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:56.133874 32238 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:56.210373 30756 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:56.186177 442 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:56.179430 425 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:56.179622 32672 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:56.179370 32574 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.209085 346 consensus_queue.cc:1035] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
I20260806 20:15:56.193902 439 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.167491 32664 consensus_queue.cc:237] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.258925 505 consensus_queue.cc:1046] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:15:56.260058 32443 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.262745 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.274907 30703 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:56.236145 32238 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.265816 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.267537 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.315181 30708 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:56.261302 30713 raft_consensus.cc:1226] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:56.238894 30602 catalog_manager.cc:5495] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: config changed from index -1 to 2, term changed from 1 to 4, NON_VOTER 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) added. New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20260806 20:15:56.268393 32574 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:56.282184 489 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:56.262768 32741 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:56.282811 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.233639 30786 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:15:56.284142 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.290385 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.292315 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.293503 32667 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:56.282811 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.272301 387 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:56.297379 313 raft_consensus.cc:3032] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:56.300223 350 raft_consensus.cc:3037] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:56.302620 351 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:56.310853 503 log.cc:924] Time spent T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.185s user 0.001s sys 0.000s
W20260806 20:15:56.312381 502 log.cc:924] Time spent T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.242s user 0.000s sys 0.001s
I20260806 20:15:56.318814 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.281224 30871 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:56.315333 30864 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:56.313587 32443 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.331739 30865 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:56.333987 30602 catalog_manager.cc:5495] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 4 to 5, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 5 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.351445 30676 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:56.352238 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.337513 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.343847 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:56.337028 387 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:56.352630 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.335247 389 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:56.366447 30864 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:56.340049 313 raft_consensus.cc:729] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:56.370529 30871 raft_consensus.cc:2379] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:56.371050 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.352276 30791 raft_consensus.cc:2379] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:56.371361 30713 raft_consensus.cc:1261] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:56.365862 30703 raft_consensus.cc:2354] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5.
I20260806 20:15:56.372030 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:56.343547 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:56.381938 30602 catalog_manager.cc:5495] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 3 to 4, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.389169 30708 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:15:56.375914 30863 raft_consensus.cc:2396] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 20:15:56.371534 387 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:56.394461 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.394702 30711 raft_consensus.cc:1226] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:56.396116 30602 catalog_manager.cc:5495] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 3, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 3 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:15:56.403247 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.347388 351 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.404661 30871 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 7
I20260806 20:15:56.384554 32733 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:56.402740 30602 catalog_manager.cc:5495] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 2 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.409548 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:15:56.402058 421 consensus_meta.cc:287] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.548s user 0.010s sys 0.000s
W20260806 20:15:56.414134 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 210071us
I20260806 20:15:56.401489 32672 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.411257 31414 raft_consensus.cc:2379] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:56.429661 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.407855 32733 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:15:56.431290 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 233983us
I20260806 20:15:56.436362 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.438300 32672 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.443878 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:56.446306 30676 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.433467 30602 catalog_manager.cc:5495] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 4, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.425581 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.452425 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.443965 32672 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:56.454869 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:56.451869 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.456600 30791 raft_consensus.cc:2379] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:56.457581 346 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:15:56.463052 512 log.cc:924] Time spent T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260806 20:15:56.457859 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.466305 30713 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:56.463403 30707 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:56.442330 32733 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:56.411579 390 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.470508 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:56.476167 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:56.476902 30866 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:56.450862 405 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.490918 30871 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:15:56.462935 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:56.493140 30789 raft_consensus.cc:2379] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:56.454293 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:56.472162 424 consensus_meta.cc:287] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.589s user 0.005s sys 0.005s
I20260806 20:15:56.447125 439 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:56.463150 346 raft_consensus.cc:686] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:56.463773 32238 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:56.503145 31415 raft_consensus.cc:2379] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:56.465507 32445 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:56.504014 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:56.492713 426 consensus_meta.cc:287] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.537s user 0.010s sys 0.000s
I20260806 20:15:56.494843 489 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.510706 30711 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:56.515275 30871 raft_consensus.cc:2379] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:56.507089 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.514104 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.517860 30602 catalog_manager.cc:5495] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 3, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 3 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.508462 30863 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:56.526870 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.526829 32238 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:15:56.532447 30708 raft_consensus.cc:1261] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260806 20:15:56.532593 30870 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.025s sys 0.063s
W20260806 20:15:56.532996 30870 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s user 0.025s sys 0.063s
I20260806 20:15:56.517884 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.511956 439 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:56.447451 313 consensus_queue.cc:260] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.528658 30868 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:56.513473 352 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.549597 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.550280 30871 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:56.551645 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.500422 405 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.554760 30756 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:56.565375 30756 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:56.571369 30602 catalog_manager.cc:5495] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:56.573434 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.549067 32741 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.553884 385 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:56.573627 507 log.cc:924] Time spent T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260806 20:15:56.604199 30871 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:56.603379 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.581329 32640 consensus_queue.cc:1046] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260806 20:15:56.577943 32274 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.613011 32274 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.615322 32274 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:56.570994 377 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:56.575800 313 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:56.570458 351 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:56.568835 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.619760 30602 catalog_manager.cc:5495] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 4, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:15:56.572714 314 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:56.621287 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.621927 30871 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:56.540783 489 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.622893 30712 raft_consensus.cc:1261] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:56.540068 439 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.633800 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.634470 30708 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:56.635264 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.635824 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.635279 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:56.636384 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.631150 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.638052 30705 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:56.625291 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.633721 439 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:56.625803 390 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:56.544205 352 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.642292 31419 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 20:15:56.643183 31423 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:56.642241 30866 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:56.625552 346 consensus_queue.cc:237] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.627921 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.644814 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.636392 30712 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:56.646343 30673 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:56.647190 32664 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:15:56.649438 32640 raft_consensus.cc:3032] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:15:56.649916 32640 raft_consensus.cc:729] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:56.650241 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.631085 32642 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.650938 30868 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 2.
I20260806 20:15:56.650816 32640 consensus_queue.cc:260] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.652030 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.651501 30602 catalog_manager.cc:5495] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 2, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 2 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:56.652766 30868 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:56.655413 32642 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:56.657056 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 223404us
I20260806 20:15:56.629773 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.630396 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.630661 489 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:56.641211 30864 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:15:56.631335 30861 raft_consensus.cc:2396] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 20:15:56.623848 314 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:56.600015 401 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.623230 385 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.623165 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.658506 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:56.660334 31389 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:56.661145 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:15:56.623677 32647 consensus_meta.cc:287] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.805s user 0.009s sys 0.003s
I20260806 20:15:56.661051 442 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.662052 421 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.645591 30866 raft_consensus.cc:2379] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:15:56.662807 509 log.cc:924] Time spent T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
W20260806 20:15:56.663477 511 log.cc:924] Time spent T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260806 20:15:56.603416 32741 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.607692 32756 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:15:56.647555 32664 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:56.723646 32664 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:56.725515 30712 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:15:56.679138 439 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260806 20:15:56.726529 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.727662 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.728642 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.664389 390 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:15:56.628942 339 consensus_meta.cc:287] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.574s user 0.007s sys 0.004s
I20260806 20:15:56.730185 339 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:56.731215 515 log.cc:924] Time spent T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20260806 20:15:56.683475 314 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:56.732666 339 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:56.733784 30863 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:56.734212 30783 raft_consensus.cc:2379] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:56.734935 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.735848 30863 raft_consensus.cc:2379] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:56.736665 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.737416 30863 raft_consensus.cc:2379] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:56.738431 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.741006 32756 raft_consensus.cc:686] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:56.741264 405 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:56.742096 32756 consensus_queue.cc:237] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.743062 337 raft_consensus.cc:3032] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:56.743460 337 raft_consensus.cc:729] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:56.744261 337 consensus_queue.cc:260] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.746888 352 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:56.750123 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.753166 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:15:56.754752 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 94506.2us
I20260806 20:15:56.755543 31389 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.658515 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.649657 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.757869 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.671452 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.609305 516 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:56.664062 30786 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:56.664316 32443 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:56.683787 30707 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:56.676718 30673 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.685251 30865 raft_consensus.cc:2362] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7. Re-sending same reply.
I20260806 20:15:56.636145 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:56.706632 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.771083 30711 raft_consensus.cc:2379] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:56.676985 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.772096 30708 raft_consensus.cc:2362] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6. Re-sending same reply.
W20260806 20:15:56.773307 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 96712.6us
I20260806 20:15:56.775098 30705 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:56.687942 30713 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:56.683723 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.776386 30864 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:15:56.706372 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.675858 30871 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:56.718358 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:56.779332 30866 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:56.694344 30710 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:56.715974 424 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:56.710407 318 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.721915 32640 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:56.767050 30869 raft_consensus.cc:2379] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:56.787184 30834 leader_election.cc:398] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5.
W20260806 20:15:56.870065 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 291658us
I20260806 20:15:56.698329 377 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.872725 30712 raft_consensus.cc:1261] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:15:56.878993 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:56.881004 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 125549us
I20260806 20:15:56.884284 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.885388 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.739125 30863 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:56.886587 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:56.886612 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.887279 31421 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:56.887490 30783 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:15:56.888566 30709 raft_consensus.cc:1261] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:56.766826 337 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:56.746891 32741 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:56.890264 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:56.906599 30785 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:56.882850 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.892819 30832 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:56.894428 30756 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:56.895754 30868 raft_consensus.cc:1261] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:56.908185 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1236) took 1094 ms. Trace:
I20260806 20:15:56.899976 30706 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
W20260806 20:15:56.896077 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 188326us
I20260806 20:15:56.908798 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:56.805338 391 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.016s sys 0.027s
I20260806 20:15:56.787111 439 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.912402 30785 raft_consensus.cc:2379] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:56.897832 30867 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:56.878315 400 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:56.904875 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.920109 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.901391 31389 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:56.910522 30871 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260806 20:15:56.908499 30868 rpcz_store.cc:276] 0806 20:15:55.813867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.869490 (+1055623us) service_pool.cc:227] Handling call
0806 20:15:56.895422 (+ 25932us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:56.907950 (+ 12528us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:56.908129 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:56.914902 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:56.788612 424 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:56.688431 32733 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.710980 421 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.688441 401 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.940164 30867 raft_consensus.cc:1261] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:56.940871 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1244) took 1035 ms. Trace:
I20260806 20:15:56.917572 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:56.911347 30861 raft_consensus.cc:2379] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:56.930970 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1240) took 1049 ms. Trace:
W20260806 20:15:56.930267 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 426817us
W20260806 20:15:56.912025 391 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s user 0.016s sys 0.028s
I20260806 20:15:56.941198 30867 rpcz_store.cc:276] 0806 20:15:55.905924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.914506 (+1008582us) service_pool.cc:227] Handling call
0806 20:15:56.939814 (+ 25308us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:56.940621 (+ 807us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:56.940812 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:56.951421 30835 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:56.905710 513 consensus_queue.cc:1046] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260806 20:15:56.878018 32507 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.937820 30834 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.911463 31417 raft_consensus.cc:2379] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:56.939881 30864 raft_consensus.cc:1226] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:15:56.971401 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1242) took 1089 ms. Trace:
I20260806 20:15:56.730121 390 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.698710 32647 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:56.946085 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.949165 400 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:15:56.957379 30866 raft_consensus.cc:1261] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:15:56.988581 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1245) took 1060 ms. Trace:
I20260806 20:15:56.950503 30871 rpcz_store.cc:276] 0806 20:15:55.881385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.902493 (+1021108us) service_pool.cc:227] Handling call
0806 20:15:56.902849 (+ 356us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:56.930657 (+ 27808us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:56.930905 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:56.961928 30869 raft_consensus.cc:1226] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:56.990036 30676 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:56.926083 32445 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:56.968523 30865 raft_consensus.cc:1261] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:15:56.992561 30676 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:56.944770 350 consensus_meta.cc:287] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.594s user 0.010s sys 0.000s
I20260806 20:15:56.936427 424 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:56.903185 318 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.971789 30864 rpcz_store.cc:276] 0806 20:15:55.881981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.913188 (+1031207us) service_pool.cc:227] Handling call
0806 20:15:56.913537 (+ 349us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:56.971106 (+ 57569us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:56.971337 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:56.984396 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1246) took 1037 ms. Trace:
I20260806 20:15:56.933859 31389 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:56.916236 524 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:56.928000 377 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.948053 434 consensus_queue.cc:1046] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260806 20:15:56.950034 489 consensus_queue.cc:1035] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.043s
I20260806 20:15:56.994238 31422 raft_consensus.cc:2379] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:56.973997 32238 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.000428 30868 rpcz_store.cc:276] 0806 20:15:55.949866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.932936 (+983070us) service_pool.cc:227] Handling call
0806 20:15:56.961262 (+ 28326us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:56.975360 (+ 14098us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:56.975391 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:56.984073 (+ 8682us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:56.984132 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:56.984203 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:56.984330 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:56.990222 400 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:56.927739 439 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:56.981237 391 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:15:56.870148 516 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:57.013463 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 79678.6us
I20260806 20:15:56.988914 30866 rpcz_store.cc:276] 0806 20:15:55.928243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.918335 (+990092us) service_pool.cc:227] Handling call
0806 20:15:56.949589 (+ 31254us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:56.988322 (+ 38733us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:56.988523 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:57.035243 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:15:57.010003 524 raft_consensus.cc:686] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:57.017204 30864 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:15:56.930016 32741 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:56.986196 32507 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:15:57.023722 30865 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:15:56.985950 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.076280 31423 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:15:57.022017 352 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:57.023110 389 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:57.088814 30712 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:57.090799 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:57.091419 30710 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:15:57.030853 400 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:57.093016 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:57.010819 32568 consensus_queue.cc:1046] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260806 20:15:57.095068 30867 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:57.097710 30865 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:15:57.094947 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:57.087625 352 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.064136 32741 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.002569 32445 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.100281 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.103991 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.085620 442 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.082382 339 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:57.108564 30709 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:57.105069 30861 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 20:15:57.115792 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 123315us
I20260806 20:15:57.033131 32664 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.117794 30676 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.086889 385 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:57.094931 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:57.119716 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.105619 352 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.106621 32741 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.114475 32238 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.141649 30703 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
W20260806 20:15:57.120101 532 log.cc:924] Time spent T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260806 20:15:57.109301 32733 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:57.161698 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:57.162389 30709 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:57.163031 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:57.099155 389 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.174360 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.136226 32638 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:57.136355 391 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:57.148589 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:57.191952 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.152400 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1251) took 1107 ms. Trace:
I20260806 20:15:57.183497 30710 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:57.158455 421 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.157742 505 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:57.195196 30712 raft_consensus.cc:2354] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6.
I20260806 20:15:57.142158 351 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260806 20:15:57.158149 32274 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:57.119822 339 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.194473 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:57.135870 524 consensus_queue.cc:237] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.210379 31388 leader_election.cc:398] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6.
I20260806 20:15:57.210767 31388 leader_election.cc:401] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:15:57.188275 32647 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.217608 32638 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:57.218217 32638 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.220256 32638 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:57.221827 527 log.cc:924] Time spent T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
W20260806 20:15:57.222472 387 consensus_meta.cc:287] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.817s user 0.019s sys 0.000s
I20260806 20:15:57.222885 387 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.225503 30786 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:15:57.196583 30861 rpcz_store.cc:276] 0806 20:15:56.041794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.989876 (+948082us) service_pool.cc:227] Handling call
0806 20:15:57.066806 (+ 76930us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:57.119365 (+ 52559us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:57.148915 (+ 29550us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:57.226177 30786 raft_consensus.cc:595] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:57.226604 377 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.229609 32274 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.231747 32274 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:15:57.232789 537 log.cc:924] Time spent T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20260806 20:15:57.236635 542 log.cc:924] Time spent T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260806 20:15:57.238687 389 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:57.239444 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:57.241776 32507 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.243016 32664 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:57.243359 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1232) took 1499 ms. Trace:
I20260806 20:15:57.202215 390 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.244335 387 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.244616 32510 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:57.194538 32733 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.245041 32443 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:57.245502 339 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.246122 30783 raft_consensus.cc:2379] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:15:57.246968 529 log.cc:924] Time spent T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20260806 20:15:57.249083 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:57.306787 31423 raft_consensus.cc:2379] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:57.250120 513 raft_consensus.cc:3032] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:57.254081 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:57.255682 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1371) took 1084 ms. Trace:
I20260806 20:15:57.246944 434 consensus_queue.cc:1225] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer 42cabca48db7403b8523a3382a046043 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260806 20:15:57.243994 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 306244us
I20260806 20:15:57.312577 30834 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:57.243424 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 292078us
W20260806 20:15:57.203068 521 log.cc:924] Time spent T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.218s user 0.001s sys 0.001s
I20260806 20:15:57.257663 548 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:57.259987 30712 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260806 20:15:57.261231 31422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 400) took 1439 ms (client timeout 1617 ms). Trace:
I20260806 20:15:57.261802 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.262559 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:57.263384 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.320744 30713 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:57.320755 30705 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 6
I20260806 20:15:57.267354 32672 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:15:57.267951 516 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.271747 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1239) took 1390 ms. Trace:
I20260806 20:15:57.272795 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1250) took 1242 ms. Trace:
I20260806 20:15:57.273777 31417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 405) took 1111 ms. Trace:
I20260806 20:15:57.336148 31417 rpcz_store.cc:276] 0806 20:15:56.162365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.162642 (+ 277us) service_pool.cc:227] Handling call
0806 20:15:57.273542 (+1110900us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b5400570b54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:57.273709 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1359002880}
I20260806 20:15:57.274931 32741 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.337623 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.338479 31389 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.338653 32741 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:57.338769 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.339272 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.339993 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.340763 30791 raft_consensus.cc:2396] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:57.263264 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1237) took 1425 ms. Trace:
I20260806 20:15:57.207609 505 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.341876 32507 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:57.342401 32507 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.343595 505 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:57.277694 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 63464.4us
I20260806 20:15:57.344322 32507 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.345582 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.281364 30707 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:15:57.346297 30787 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:57.226562 30786 raft_consensus.cc:603] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:57.292198 30709 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:15:57.292624 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1376) took 1033 ms. Trace:
I20260806 20:15:57.347548 30834 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:57.200800 401 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.243829 30870 rpcz_store.cc:276] 0806 20:15:55.744406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:55.830850 (+ 86444us) service_pool.cc:227] Handling call
0806 20:15:55.852036 (+ 21186us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:55.876380 (+ 24344us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:15:55.892504 (+ 16124us) log.cc:842] Serialized 14 byte log entry
0806 20:15:56.939333 (+1046829us) log.cc:842] Serialized 14 byte log entry
0806 20:15:57.242854 (+303521us) spinlock_profiling.cc:243] Waited 8.21 ms on lock 0x7b180041b720. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f714e58e990 00007f714e750864 00007f714e750155 00007f71516673d0 00007f7151679576 00007f715168a0d7 00007f715168a058
0806 20:15:57.243016 (+ 162us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:57.243063 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:57.243103 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:57.243178 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:57.243279 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28480,"prepare.queue_time_us":39057,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":84,"raft.queue_time_us":338949,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":8729,"spinlock_wait_cycles":18877184,"thread_start_us":953885,"threads_started":2,"wal-append.queue_time_us":590656}
I20260806 20:15:57.276461 439 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.349104 30868 rpcz_store.cc:276] 0806 20:15:56.259590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.023163 (+763573us) service_pool.cc:227] Handling call
0806 20:15:57.023500 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:57.023731 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:15:57.069261 (+ 45530us) log.cc:842] Serialized 14 byte log entry
0806 20:15:57.255510 (+186249us) log.cc:842] Serialized 14 byte log entry
0806 20:15:57.292288 (+ 36778us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:57.292341 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:57.292384 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:57.292448 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:57.292559 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74990,"prepare.queue_time_us":46126,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":67642,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":161,"thread_start_us":150414,"threads_started":2,"wal-append.queue_time_us":193474}
I20260806 20:15:57.305869 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.354311 30706 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:57.274866 32642 raft_consensus.cc:3032] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:15:57.285662 31414 raft_consensus.cc:2379] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:57.275905 32445 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:57.309814 30871 rpcz_store.cc:276] 0806 20:15:56.171321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.998161 (+826840us) service_pool.cc:227] Handling call
0806 20:15:56.998507 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:56.998732 (+ 225us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:15:57.022668 (+ 23936us) log.cc:842] Serialized 14 byte log entry
0806 20:15:57.255220 (+232552us) log.cc:842] Serialized 14 byte log entry
0806 20:15:57.255353 (+ 133us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:57.255391 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:57.255431 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:57.255494 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:57.255626 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":102087,"prepare.queue_time_us":32050,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":69,"raft.queue_time_us":103628,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":155,"thread_start_us":130050,"threads_started":1}
W20260806 20:15:57.312696 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 45360.4us
I20260806 20:15:57.319207 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:15:57.324786 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 125591us
W20260806 20:15:57.319873 31422 rpcz_store.cc:269] 0806 20:15:55.821680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:55.821892 (+ 212us) service_pool.cc:227] Handling call
0806 20:15:57.260994 (+1439102us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540067f2d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:57.261141 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2624506240}
I20260806 20:15:57.326326 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.332252 30867 rpcz_store.cc:276] 0806 20:15:56.030059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.951560 (+921501us) service_pool.cc:227] Handling call
0806 20:15:56.951835 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:57.102220 (+150385us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:57.272538 (+170318us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b5400376c54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:57.272730 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":318497536}
W20260806 20:15:57.275815 314 consensus_meta.cc:287] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.543s user 0.007s sys 0.007s
I20260806 20:15:57.350148 31388 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.342137 30869 rpcz_store.cc:276] 0806 20:15:55.838400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.888553 (+1050153us) service_pool.cc:227] Handling call
0806 20:15:56.955164 (+ 66611us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:56.990661 (+ 35497us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:57.263050 (+272389us) spinlock_profiling.cc:243] Waited 66.3 ms on lock 0x7b540019da50. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:57.263200 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152513280}
I20260806 20:15:57.294696 318 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.356885 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:57.309633 513 raft_consensus.cc:729] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:57.317688 32510 raft_consensus.cc:686] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:57.367328 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.368350 31388 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:15:57.369069 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 251358us
I20260806 20:15:57.364264 31421 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:15:57.347604 535 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:57.368788 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:57.377414 30754 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:57.384603 31423 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.353765 32757 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:57.376374 30709 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:57.377554 30756 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:57.357791 32672 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:57.393559 30705 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:57.360512 530 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:57.383118 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:57.404071 30785 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.356859 32642 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:57.361989 442 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:57.399222 30754 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:57.382069 30834 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.371428 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:57.401269 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.406394 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:57.406953 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:57.410878 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:57.423825 30787 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:15:57.410771 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:57.383585 539 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:57.413575 533 raft_consensus.cc:3032] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:57.414710 30676 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.418462 30756 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:57.430020 30835 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:57.421515 30867 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:15:57.373956 32647 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:57.374464 434 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:57.424010 30869 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:57.423960 30871 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
W20260806 20:15:57.412467 544 log.cc:924] Time spent T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.197s user 0.001s sys 0.001s
I20260806 20:15:57.420022 31423 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:15:57.327929 548 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:57.427790 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 55666.3us
I20260806 20:15:57.441788 30757 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:15:57.430466 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 48503.1us
I20260806 20:15:57.366781 32668 consensus_queue.cc:1035] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260806 20:15:57.389683 426 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:57.332031 30863 rpcz_store.cc:276] 0806 20:15:55.881037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:56.886854 (+1005817us) service_pool.cc:227] Handling call
0806 20:15:56.887187 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:57.111320 (+224133us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:57.117045 (+ 5725us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:57.119851 (+ 2806us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:15:57.125157 (+ 5306us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:57.271456 (+146299us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540019cb54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:57.271508 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:57.271666 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":465568512}
I20260806 20:15:57.428076 32756 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:57.440811 30676 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:57.426402 351 raft_consensus.cc:2781] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:15:57.443969 30832 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:57.373531 314 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.439497 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:57.439785 555 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.443500 439 raft_consensus.cc:2726] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:57.455391 32507 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:15:57.455871 516 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:57.439525 543 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:57.481217 30870 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0"
dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47072
I20260806 20:15:57.457122 30834 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.460904 31418 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 20:15:57.461972 552 log.cc:924] Time spent T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20260806 20:15:57.441099 539 raft_consensus.cc:686] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:57.433017 30861 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:57.462710 489 consensus_queue.cc:1035] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:57.464022 30783 raft_consensus.cc:1261] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:15:57.458644 351 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.431056 533 raft_consensus.cc:729] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:57.500104 314 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:57.402763 32733 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260806 20:15:57.500567 30834 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.402354 350 consensus_queue.cc:1046] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260806 20:15:57.470276 30708 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:57.501914 30834 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.503055 30834 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.503284 489 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:57.504136 30834 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.504318 351 raft_consensus.cc:3037] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:57.493359 346 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.491727 543 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.506127 346 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.506762 346 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:15:57.507431 346 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.508543 543 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.393209 535 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:57.510558 313 consensus_meta.cc:287] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.865s user 0.006s sys 0.006s
I20260806 20:15:57.484076 32642 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.511986 32642 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:57.512802 30868 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:57.487351 516 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.467382 426 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.518838 548 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.519556 516 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.482111 32568 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.038s
I20260806 20:15:57.520159 426 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:57.521274 551 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.171s user 0.000s sys 0.001s
W20260806 20:15:57.522718 424 consensus_meta.cc:287] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.501s user 0.010s sys 0.002s
I20260806 20:15:57.523130 424 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.524791 31414 raft_consensus.cc:1226] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3. Ops: []
I20260806 20:15:57.474376 30756 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:57.452852 505 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 20:15:57.525856 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 51544.2us
I20260806 20:15:57.526252 505 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:15:57.527009 505 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.527624 505 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:57.457548 427 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 6
I20260806 20:15:57.529750 30869 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:57.489249 32510 consensus_queue.cc:237] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.532003 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:57.526607 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:57.534080 314 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:15:57.481880 30870 raft_consensus.cc:595] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:15:57.534565 30870 raft_consensus.cc:603] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:57.457406 389 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:15:57.456845 32756 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.485577 30788 raft_consensus.cc:2379] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:57.538794 314 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.458144 32647 raft_consensus.cc:686] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:57.541733 314 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:57.542378 32647 consensus_queue.cc:237] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:57.533215 558 log.cc:924] Time spent T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20260806 20:15:57.544123 514 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:15:57.547283 30869 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bf7304ca0290443b884297d0e8744607"
dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47072
I20260806 20:15:57.547912 30869 raft_consensus.cc:595] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:15:57.548250 30869 raft_consensus.cc:603] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:15:57.474285 550 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:57.465284 549 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.099s
I20260806 20:15:57.551745 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.541247 389 raft_consensus.cc:686] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:57.552719 550 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:57.553207 30783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec226f72b38a49529b88339beb36af46"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:15:57.553011 389 consensus_queue.cc:237] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.499724 533 consensus_queue.cc:260] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.466872 30757 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:57.522209 400 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.481488 31420 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:15:57.460219 524 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.534164 30756 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:57.531772 30676 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:57.533186 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:57.403513 32672 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.537719 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.485954 434 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:57.549881 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:57.577629 30866 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
W20260806 20:15:57.580644 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 48979.7us
W20260806 20:15:57.580683 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 103731us
I20260806 20:15:57.581467 30676 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:57.553928 30783 raft_consensus.cc:595] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:57.482589 32507 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.495877 539 consensus_queue.cc:237] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.525220 424 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.588845 30676 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:57.566161 32568 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:15:57.566390 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 99600.8us
I20260806 20:15:57.558801 550 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:15:57.550621 32640 consensus_meta.cc:287] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.662s user 0.004s sys 0.012s
I20260806 20:15:57.559068 30785 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
W20260806 20:15:57.579120 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 45038.1us
I20260806 20:15:57.541539 32756 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:57.581828 31419 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:57.557775 535 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.607650 30756 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:57.607651 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 103572us
I20260806 20:15:57.582922 328 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:57.613600 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:57.615685 30834 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.615351 30756 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:57.617189 30834 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:57.589231 32507 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:57.616146 32733 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:57.548369 513 consensus_queue.cc:260] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:57.560649 352 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.603142 32757 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:57.622087 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.623754 30709 raft_consensus.cc:1261] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:15:57.603550 434 raft_consensus.cc:686] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:57.625798 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.580024 549 consensus_queue.cc:1046] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260806 20:15:57.633100 30834 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.636957 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.637413 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.616590 328 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:57.625594 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:57.672910 30713 raft_consensus.cc:2379] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:57.662549 30790 raft_consensus.cc:1261] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260806 20:15:57.670110 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 43973us
I20260806 20:15:57.673547 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.588150 514 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:57.591493 489 raft_consensus.cc:686] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:57.663432 30708 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:15:57.629837 31389 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.630852 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:57.676186 562 log.cc:924] Time spent T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260806 20:15:57.700297 30834 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:57.618814 313 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:15:57.700318 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 70568.4us
I20260806 20:15:57.702095 30834 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.676971 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:57.706634 543 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.706873 30786 raft_consensus.cc:2379] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:57.621803 32733 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.707715 32733 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:57.709753 564 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.710027 535 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:15:57.711369 30756 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:57.683676 337 consensus_meta.cc:287] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.770s user 0.006s sys 0.010s
I20260806 20:15:57.683271 31416 raft_consensus.cc:2379] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:57.713995 434 consensus_queue.cc:237] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.715128 30867 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b"
dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47072
I20260806 20:15:57.715744 30867 raft_consensus.cc:595] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:15:57.679229 350 consensus_queue.cc:1035] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:15:57.676407 549 raft_consensus.cc:3032] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:15:57.620271 32741 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.615092 32668 consensus_queue.cc:1035] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260806 20:15:57.719748 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:57.718148 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 107159us
I20260806 20:15:57.721855 30713 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.720384 30834 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.735086 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.629118 533 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.760286 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:57.766276 30708 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:57.629168 32640 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:57.729660 30757 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:57.716922 352 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.700892 32757 raft_consensus.cc:686] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:57.685490 328 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:15:57.770923 570 log.cc:924] Time spent T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260806 20:15:57.777771 30754 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:57.797813 30676 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.772395 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.721303 549 raft_consensus.cc:729] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:57.771736 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.755384 564 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:57.773252 427 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6.
I20260806 20:15:57.769364 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.727164 555 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:15:57.699419 514 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.810704 30703 raft_consensus.cc:2396] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:15:57.789065 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.812584 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:57.790164 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 86632us
I20260806 20:15:57.792631 30713 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:57.772989 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.814483 31388 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:57.814548 31389 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.759552 535 consensus_queue.cc:1035] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260806 20:15:57.727684 505 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:57.796547 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 66984.7us
I20260806 20:15:57.774259 32638 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260806 20:15:57.724155 513 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.710104 400 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:57.721752 313 raft_consensus.cc:686] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:57.725750 388 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:15:57.807063 30787 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:15:57.637264 426 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:57.779443 32640 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.916749 30708 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:15:57.787468 567 consensus_queue.cc:1035] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.101s
I20260806 20:15:57.892884 30703 raft_consensus.cc:1261] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:15:57.841069 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:57.823309 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:57.863770 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:57.799300 387 raft_consensus.cc:2726] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:57.699743 530 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:57.920653 30711 raft_consensus.cc:2379] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:57.921622 31388 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:15:57.809203 389 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.869151 32672 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:57.823398 32238 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:57.994037 30835 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:57.879253 571 log.cc:924] Time spent T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.204s user 0.001s sys 0.000s
I20260806 20:15:57.937783 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:15:57.842365 518 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:57.812387 514 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.808120 555 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:57.994616 30703 raft_consensus.cc:1261] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 20:15:57.996531 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 72716.5us
W20260806 20:15:57.998303 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 183848us
I20260806 20:15:58.014801 31389 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:57.999392 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:15:57.851572 427 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:57.924124 32443 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:57.992738 400 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:15:57.923641 32640 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.808182 548 consensus_queue.cc:1035] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.141s
I20260806 20:15:57.932543 439 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:58.017792 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 196489us
I20260806 20:15:57.923861 32668 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:57.945494 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:57.994105 401 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:57.888105 314 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:57.994818 426 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:57.997781 530 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:58.035439 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.003315 388 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:58.052657 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.053375 30867 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:58.016583 555 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:57.811692 489 consensus_queue.cc:237] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.054422 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.013311 32238 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:58.055886 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:57.868038 549 consensus_queue.cc:260] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:58.077050 30834 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.058681 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:58.059113 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.050697 424 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:15:58.063612 30869 raft_consensus.cc:2354] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6.
I20260806 20:15:58.097780 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.057096 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.070914 32238 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.073792 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:57.912645 32757 consensus_queue.cc:237] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:58.058182 30676 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:58.014405 538 consensus_queue.cc:1035] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260806 20:15:58.047116 535 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:58.081626 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:58.028887 32443 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.026000 337 consensus_queue.cc:1035] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260806 20:15:58.047559 400 raft_consensus.cc:686] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:58.060308 30791 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:58.088084 30834 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.118898 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:58.099385 30871 raft_consensus.cc:2379] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:58.086439 30865 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:58.120611 30834 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:58.100952 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 86876.5us
I20260806 20:15:58.101387 30868 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:15:58.026566 32733 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.128777 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.083972 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:58.129621 30867 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:58.128849 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.019469 518 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.118916 30864 raft_consensus.cc:2379] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:58.133383 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.047945 32668 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:58.120285 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 62202.5us
I20260806 20:15:58.123929 30834 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.049465 439 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.096675 576 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:58.103372 32647 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:15:58.051808 426 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:58.060279 388 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.129647 568 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:58.155836 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.139397 30787 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:58.138916 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.129168 32510 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:58.056385 530 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.107973 535 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:58.168066 30869 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:15:58.175527 30867 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
W20260806 20:15:58.177240 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 141894us
I20260806 20:15:58.178076 31389 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:58.149559 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.150805 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:58.152246 505 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.156620 30870 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:15:58.042951 313 consensus_queue.cc:237] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.129736 578 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:15:58.184856 351 consensus_meta.cc:287] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.680s user 0.011s sys 0.002s
I20260806 20:15:58.144300 32638 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:58.120059 385 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20260806 20:15:58.161477 30865 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:58.180593 30706 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:58.152350 30709 raft_consensus.cc:2379] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:58.158494 32445 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:58.170665 535 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:58.170915 568 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:15:58.219575 30867 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260806 20:15:58.248150 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 70093.7us
I20260806 20:15:58.248837 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:58.216367 586 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:58.268081 31389 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.265136 30868 raft_consensus.cc:1261] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:15:58.268345 30711 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
W20260806 20:15:58.254210 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 96024.3us
I20260806 20:15:58.220230 424 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 20:15:58.269387 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 111383us
I20260806 20:15:58.247866 578 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:58.275702 30708 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:58.218333 30783 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:58.261292 425 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:58.279505 590 consensus_queue.cc:1046] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260806 20:15:58.286696 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:58.270725 574 log.cc:924] Time spent T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.228s user 0.001s sys 0.000s
I20260806 20:15:58.248339 337 consensus_queue.cc:1035] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260806 20:15:58.274880 505 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:58.309181 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:58.285753 32642 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:58.300834 32238 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.262938 32638 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.252879 576 consensus_queue.cc:1035] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
I20260806 20:15:58.255684 351 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:58.279323 578 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:58.290392 314 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:58.320994 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:58.317041 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.303438 424 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:15:58.266918 32445 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.221092 400 consensus_queue.cc:237] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:58.327961 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:58.297518 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:15:58.348708 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:58.311583 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 256797us
I20260806 20:15:58.351052 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:58.351693 30867 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:15:58.323431 580 log.cc:924] Time spent T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20260806 20:15:58.370152 31389 leader_election.cc:398] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6.
I20260806 20:15:58.371057 31389 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:15:58.358434 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:58.372238 31389 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:58.354074 30792 raft_consensus.cc:2354] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3.
W20260806 20:15:58.364265 572 log.cc:924] Time spent T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.283s user 0.002s sys 0.000s
I20260806 20:15:58.365401 351 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:58.367046 439 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:58.311266 32574 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260806 20:15:58.367691 32638 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.322921 32642 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.328745 576 consensus_queue.cc:1225] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer 42cabca48db7403b8523a3382a046043 log is divergent from this leader: its last log entry 2.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260806 20:15:58.350586 32507 consensus_meta.cc:287] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.688s user 0.013s sys 0.000s
W20260806 20:15:58.367208 582 log.cc:924] Time spent T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20260806 20:15:58.300232 425 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.329531 32733 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.402397 30754 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:58.451795 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 49610.1us
I20260806 20:15:58.328179 32443 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:58.404201 30708 raft_consensus.cc:2379] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:15:58.369000 426 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:58.354277 32668 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:58.318483 505 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.420147 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:58.453370 30787 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:15:58.450024 586 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:58.415310 577 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.252s user 0.001s sys 0.000s
I20260806 20:15:58.460628 424 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:15:58.467636 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 86439.2us
I20260806 20:15:58.455286 351 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:15:58.401643 388 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.398942 518 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.447707 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:15:58.536448 592 log.cc:924] Time spent T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Append to log took a long time: real 0.066s user 0.000s sys 0.003s
I20260806 20:15:58.413558 32507 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.474696 586 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.616046 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:58.545810 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:58.456673 530 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:58.533133 30713 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.007s sys 0.008s
W20260806 20:15:58.733793 30713 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s user 0.008s sys 0.008s
W20260806 20:15:58.705389 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 68111.7us
I20260806 20:15:58.620066 30706 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:15:58.736125 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:58.737880 30706 raft_consensus.cc:2379] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:58.605538 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:58.718616 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:58.559010 31415 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:15:58.448272 427 raft_consensus.cc:686] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:15:58.738823 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:58.741931 506 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:58.742514 506 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.743772 586 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:58.744508 506 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.745019 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:58.745692 30707 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:15:58.746562 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:58.747169 32507 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:58.747946 513 consensus_meta.cc:287] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.816s user 0.008s sys 0.003s
I20260806 20:15:58.748401 568 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:58.748939 568 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:58.704105 391 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:58.749994 391 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.751001 568 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:58.752490 30863 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:15:58.752465 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:58.561654 426 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:58.703949 547 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.000s sys 0.015s
I20260806 20:15:58.741423 427 consensus_queue.cc:237] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:58.753868 426 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:58.755191 530 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:58.755903 426 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:58.755751 530 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:58.561038 31414 raft_consensus.cc:3037] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:15:58.752810 328 consensus_meta.cc:287] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.946s user 0.009s sys 0.003s
I20260806 20:15:58.757818 328 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.758895 30786 raft_consensus.cc:1226] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:15:58.759923 328 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:58.761034 547 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.719s user 0.003s sys 0.018s
W20260806 20:15:58.737476 607 log.cc:924] Time spent T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260806 20:15:58.762779 30791 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:15:58.655284 30867 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
W20260806 20:15:58.610133 31417 consensus_meta.cc:287] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.768s user 0.012s sys 0.000s
I20260806 20:15:58.765069 30788 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:15:58.763402 31417 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:58.769404 32642 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:15:58.773101 31417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49046 (request call id 516) took 1428 ms (client timeout 1727 ms). Trace:
W20260806 20:15:58.773713 31417 rpcz_store.cc:269] 0806 20:15:57.344647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.346983 (+ 2336us) service_pool.cc:227] Handling call
0806 20:15:58.772925 (+1425942us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54005ed8d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:58.773029 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":882031360}
I20260806 20:15:58.782239 530 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:58.783913 391 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.786610 32507 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260806 20:15:58.770852 533 consensus_meta.cc:287] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.991s user 0.006s sys 0.008s
I20260806 20:15:58.772886 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:58.790539 30868 raft_consensus.cc:2396] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260806 20:15:58.792069 514 consensus_meta.cc:287] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.771s user 0.010s sys 0.001s
I20260806 20:15:58.741642 30706 raft_consensus.cc:2379] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:58.829206 30784 raft_consensus.cc:2379] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:58.792114 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:58.618767 587 log.cc:924] Time spent T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.305s user 0.000s sys 0.001s
I20260806 20:15:58.795784 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.837440 30834 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:58.557476 32756 consensus_meta.cc:287] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.940s user 0.015s sys 0.010s
W20260806 20:15:58.610880 585 log.cc:924] Time spent T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.256s user 0.000s sys 0.000s
I20260806 20:15:58.828007 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:58.842308 30834 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.794318 30712 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:15:58.551604 424 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 20:15:58.633152 575 log.cc:924] Time spent T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.229s user 0.001s sys 0.000s
I20260806 20:15:58.840229 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:58.612413 32647 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:58.613623 339 raft_consensus.cc:3032] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:15:58.844743 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1531) took 1056 ms. Trace:
I20260806 20:15:58.848440 30756 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:15:58.671564 579 log.cc:924] Time spent T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.405s user 0.001s sys 0.000s
I20260806 20:15:58.633682 615 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:15:58.634178 581 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.305s user 0.001s sys 0.000s
W20260806 20:15:58.844069 389 consensus_meta.cc:287] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.847s user 0.004s sys 0.008s
I20260806 20:15:58.364907 588 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.577s
I20260806 20:15:58.893354 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:58.835333 32445 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.860103 30713 rpcz_store.cc:276] 0806 20:15:57.788344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.817555 (+ 29211us) service_pool.cc:227] Handling call
0806 20:15:57.818400 (+ 845us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:57.892405 (+ 74005us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:57.892436 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:57.922859 (+ 30423us) log.cc:842] Serialized 26 byte log entry
0806 20:15:58.734331 (+811472us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:58.835066 (+100735us) log.cc:842] Serialized 14 byte log entry
0806 20:15:58.844413 (+ 9347us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:58.844495 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:58.844538 (+ 43us) raft_consensus.cc:1671] finished
0806 20:15:58.844550 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:58.844681 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89890,"prepare.queue_time_us":105265,"raft.queue_time_us":62618,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"thread_start_us":729088,"threads_started":1,"wal-append.queue_time_us":663033}
I20260806 20:15:58.856629 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:58.865567 615 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:58.913743 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:58.915527 30756 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:58.840457 530 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:15:58.839869 32741 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:58.861912 32647 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:58.843493 586 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260806 20:15:58.921319 524 raft_consensus.cc:979] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207: : Instructing follower 58d59beb597f4ed99d84d102ddb41ff5 to start an election
W20260806 20:15:59.002779 30757 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.014s sys 0.056s
I20260806 20:15:59.004423 589 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:58.865250 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:58.910112 32642 raft_consensus.cc:2781] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:15:58.929075 615 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:58.923877 617 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:15:59.001291 32640 consensus_meta.cc:287] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.955s user 0.008s sys 0.003s
I20260806 20:15:59.041878 609 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:58.999212 318 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:15:59.021313 31415 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 20:15:59.055176 318 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:59.055439 30832 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.859879 339 raft_consensus.cc:729] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:15:58.928933 550 consensus_meta.cc:287] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.316s user 0.010s sys 0.006s
I20260806 20:15:58.876770 549 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:15:59.010838 505 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.867838 425 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.910156 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:59.016532 560 raft_consensus.cc:3037] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:59.055474 589 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.057009 32741 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:15:59.036615 596 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:59.056068 617 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.058804 30832 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:58.849432 32756 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:59.053592 30757 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.787s user 0.014s sys 0.056s
I20260806 20:15:59.058573 339 consensus_queue.cc:260] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:59.059646 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 810821us
I20260806 20:15:59.017066 30791 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:59.060292 339 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:15:59.060837 32756 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:58.841179 514 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.061580 30785 raft_consensus.cc:2379] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:59.061547 609 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.063869 514 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:59.010762 32568 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:59.064914 30713 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:15:59.064728 32568 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:58.918430 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:59.066805 30832 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:59.056597 550 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.010179 489 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:59.060451 32741 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:59.069296 32741 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:15:59.055629 318 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:59.071070 32741 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:59.071936 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.074070 609 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.074353 550 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.054639 32640 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.078764 30789 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:15:59.058166 596 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.080655 32640 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.082355 596 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:58.926613 389 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:15:59.017256 535 consensus_meta.cc:287] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.727s user 0.008s sys 0.002s
W20260806 20:15:59.012284 30676 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.003s sys 0.010s
W20260806 20:15:59.093293 30676 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.750s user 0.003s sys 0.010s
I20260806 20:15:59.087116 589 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.068148 32568 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.084635 596 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:59.093974 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1527) took 1306 ms. Trace:
W20260806 20:15:59.105677 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 45370.7us
I20260806 20:15:59.096248 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1266) took 1035 ms. Trace:
W20260806 20:15:59.022223 30673 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.196:35519 timed out after 1.727s (SENT) [suppressed 13 similar messages]
W20260806 20:15:59.099444 585 log.cc:924] Time spent T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260806 20:15:59.100499 535 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.093534 32642 raft_consensus.cc:686] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:59.102909 31389 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:59.108747 31421 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 524) took 1523 ms. Trace:
I20260806 20:15:59.106015 30710 rpcz_store.cc:276] 0806 20:15:57.787266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.813523 (+ 26257us) service_pool.cc:227] Handling call
0806 20:15:57.813853 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:57.865857 (+ 52004us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:57.865885 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:57.866218 (+ 333us) log.cc:842] Serialized 26 byte log entry
0806 20:15:58.131662 (+265444us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:58.276396 (+144734us) log.cc:842] Serialized 14 byte log entry
0806 20:15:58.413078 (+136682us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:58.414276 (+ 1198us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:59.093762 (+679486us) raft_consensus.cc:1671] finished
0806 20:15:59.093783 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:59.093914 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31791,"prepare.queue_time_us":180731,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":98354,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":27250,"thread_start_us":401232,"threads_started":3,"wal-append.queue_time_us":354252}
I20260806 20:15:59.115856 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.117069 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.125847 30784 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:59.148502 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.128890 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:59.148506 30870 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:59.132143 30790 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:15:59.109575 589 raft_consensus.cc:1067] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Signalling peer 58d59beb597f4ed99d84d102ddb41ff5 to start an election
I20260806 20:15:59.122486 31421 rpcz_store.cc:276] 0806 20:15:57.574314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.574490 (+ 176us) service_pool.cc:227] Handling call
0806 20:15:57.715623 (+141133us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:15:57.715818 (+ 195us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:57.762379 (+ 46561us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:57.762413 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:57.800590 (+ 38177us) log.cc:842] Serialized 26 byte log entry
0806 20:15:57.964182 (+163592us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:15:57.964232 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:57.964323 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:59.090696 (+1126373us) raft_consensus.cc:1671] finished
0806 20:15:59.090723 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:59.097271 (+ 6548us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4581,"raft.queue_time_us":41820,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":22343,"spinlock_wait_cycles":40788736,"thread_start_us":148111,"threads_started":1,"wal-append.queue_time_us":242366}
I20260806 20:15:59.072058 489 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.181236 425 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:59.183064 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:59.161226 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.129753 609 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:59.185969 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.185079 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:15:59.165541 610 log.cc:924] Time spent T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260806 20:15:59.193173 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:59.167258 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 60823.7us
I20260806 20:15:59.190951 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1390) took 1434 ms. Trace:
I20260806 20:15:59.194965 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.195611 30869 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
W20260806 20:15:59.154901 555 consensus_meta.cc:287] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.080s user 0.005s sys 0.006s
I20260806 20:15:59.148607 32733 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:59.200165 30676 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:59.195329 30866 rpcz_store.cc:276] 0806 20:15:57.756958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.757105 (+ 147us) service_pool.cc:227] Handling call
0806 20:15:57.770074 (+ 12969us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:57.799150 (+ 29076us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:57.799182 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:57.806123 (+ 6941us) log.cc:842] Serialized 52 byte log entry
0806 20:15:58.057812 (+251689us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:15:58.103093 (+ 45281us) log.cc:842] Serialized 14 byte log entry
0806 20:15:58.669714 (+566621us) spinlock_profiling.cc:243] Waited 9.17 ms on lock 0x7b18002e4440. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f714e58e990 00007f714e750864 00007f714e750155 00007f7155044167 00007f7151679807 00007f7151688f0f 00007f7151688e88
0806 20:15:58.669956 (+ 242us) log.cc:842] Serialized 14 byte log entry
0806 20:15:58.761872 (+ 91916us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:58.761943 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:59.190668 (+428725us) raft_consensus.cc:1671] finished
0806 20:15:59.190698 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:59.190884 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":646469,"prepare.queue_time_us":613212,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":73,"raft.queue_time_us":130755,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":15036,"spinlock_wait_cycles":53417600,"thread_start_us":167222,"threads_started":1,"wal-append.queue_time_us":482244}
I20260806 20:15:59.115864 596 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.211392 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:59.165285 31414 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:15:59.210870 30709 pending_rounds.cc:76] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 2
I20260806 20:15:59.216995 30676 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:59.217073 30709 pending_rounds.cc:97] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.3
I20260806 20:15:59.219250 391 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:15:59.220153 32642 consensus_queue.cc:237] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.223042 535 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:59.223793 489 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:59.204283 351 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260806 20:15:59.146826 618 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.283s
I20260806 20:15:59.228296 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:59.228886 30866 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:59.230180 391 raft_consensus.cc:686] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:15:59.230679 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1343) took 1412 ms. Trace:
I20260806 20:15:59.148162 318 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.231491 30709 raft_consensus.cc:2894] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Aborting config change with OpId 2.3: Aborted: Op aborted by new leader
I20260806 20:15:59.252272 30709 raft_consensus.cc:2866] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:15:59.212169 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:59.254292 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:15:59.235882 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 169161us
I20260806 20:15:59.238420 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:59.238871 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 216673us
I20260806 20:15:59.229944 30869 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:59.188898 328 consensus_queue.cc:1035] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260806 20:15:59.266685 30754 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:59.269340 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:59.234680 545 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:59.275882 545 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.276986 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:53798 (request call id 10) took 23921 ms. Trace:
I20260806 20:15:59.277978 545 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.277375 30709 rpcz_store.cc:276] 0806 20:15:35.355881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:35.449376 (+ 93495us) service_pool.cc:227] Handling call
0806 20:15:35.468587 (+ 19211us) log.cc:842] Serialized 513 byte log entry
0806 20:15:35.623684 (+155097us) spinlock_profiling.cc:243] Waited 3.39 ms on lock 0x7b18000cbb60. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f714e58e990 00007f714e750864 00007f714e750155 00007f71515e9320 00007f715164f113 00007f71516757e2 00007f71565b1f39
0806 20:15:35.631209 (+ 7525us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b18000cbb60. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f7155517bcf 00007f714e7537b6 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d
0806 20:15:35.863106 (+231897us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b18000cbb60. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f7155517bcf 00007f714e7537b6 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d
0806 20:15:59.276896 (+23413790us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":320246,"raft.queue_time_us":201613,"raft.run_cpu_time_us":2026,"raft.run_wall_time_us":52620,"spinlock_wait_cycles":14739584,"thread_start_us":39651,"threads_started":1,"wal-append.queue_time_us":42959,"wal-append.run_cpu_time_us":1963,"wal-append.run_wall_time_us":1023812}
I20260806 20:15:59.279855 613 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:59.280408 613 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.281093 318 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.282598 613 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.283663 560 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:15:59.286554 506 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:59.287142 506 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.279633 614 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:59.290364 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:15:59.280357 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:59.192880 425 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:15:59.293282 31388 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.027s sys 0.035s
I20260806 20:15:59.302052 30676 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:15:59.313158 31388 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.784s user 0.029s sys 0.035s
I20260806 20:15:59.231423 549 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:15:59.313897 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 785228us
W20260806 20:15:59.310527 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 43928.8us
W20260806 20:15:59.321575 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 68621.9us
I20260806 20:15:59.325255 30784 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:15:59.210665 32733 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.289225 506 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.317269 30866 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:59.328612 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.202092 555 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.233232 505 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.087s
I20260806 20:15:59.290539 617 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.341435 30869 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:59.326643 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.293972 514 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:59.324021 549 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:59.367578 631 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:59.368402 608 raft_consensus.cc:2781] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:15:59.325989 613 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:15:59.370383 506 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:15:59.368460 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1387) took 1901 ms. Trace:
I20260806 20:15:59.390079 30863 rpcz_store.cc:276] 0806 20:15:57.467413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.470090 (+ 2677us) service_pool.cc:227] Handling call
0806 20:15:57.493171 (+ 23081us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:58.752984 (+1259813us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:59.368210 (+615226us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540019e454. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:59.368400 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2895699712}
I20260806 20:15:59.318408 32672 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:15:59.364826 30784 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:59.392915 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.393399 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:59.362818 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:15:59.379007 30757 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:15:59.380749 549 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:59.391747 608 raft_consensus.cc:686] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:59.399152 608 consensus_queue.cc:237] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:59.363600 572 log.cc:924] Time spent T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260806 20:15:59.381306 631 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.235944 30783 rpcz_store.cc:276] 0806 20:15:57.818406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.827267 (+ 8861us) service_pool.cc:227] Handling call
0806 20:15:59.230485 (+1403218us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54003a07d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.230609 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":312954624}
I20260806 20:15:59.402797 596 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:15:59.392009 32672 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:59.394393 506 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.397534 613 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:59.403970 631 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:59.409577 32672 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:15:59.408865 613 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:59.410281 30757 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:15:59.410579 613 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:59.409034 619 consensus_queue.cc:1046] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.050s
I20260806 20:15:59.414709 619 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:15:59.415340 619 raft_consensus.cc:3032] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:15:59.415678 619 raft_consensus.cc:729] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:15:59.408812 31417 raft_consensus.cc:2379] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:15:59.415482 506 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.293736 614 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.384102 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:15:59.419164 30655 auto_rebalancer.cc:254] failed to send replica move request: Aborted: Op aborted by new leader
I20260806 20:15:59.405200 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.403800 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.426065 30783 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:15:59.427340 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:15:59.428645 578 consensus_meta.cc:287] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.077s user 0.012s sys 0.000s
I20260806 20:15:59.327437 391 consensus_queue.cc:237] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.436988 385 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:15:59.440152 549 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.444295 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:59.417117 514 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:59.416424 425 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.429105 578 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.428668 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.377241 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.522533 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1521) took 1754 ms. Trace:
W20260806 20:15:59.584951 30785 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1336) took 1953 ms (client timeout 2189 ms). Trace:
I20260806 20:15:59.614071 30756 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:15:59.488797 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1340) took 1173 ms. Trace:
I20260806 20:15:59.107189 30871 rpcz_store.cc:276] 0806 20:15:58.061245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.142230 (+ 80985us) service_pool.cc:227] Handling call
0806 20:15:58.142612 (+ 382us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:58.254880 (+112268us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:58.254910 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:58.258687 (+ 3777us) log.cc:842] Serialized 26 byte log entry
0806 20:15:58.401249 (+142562us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:59.087758 (+686509us) log.cc:842] Serialized 14 byte log entry
0806 20:15:59.095929 (+ 8171us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:59.096013 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:59.096054 (+ 41us) raft_consensus.cc:1671] finished
0806 20:15:59.096066 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:59.096188 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":297177,"prepare.queue_time_us":344816,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":39631,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":44968,"thread_start_us":586638,"threads_started":3,"wal-append.queue_time_us":198196}
I20260806 20:15:59.570104 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1536) took 1011 ms. Trace:
I20260806 20:15:59.590816 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1690) took 1265 ms. Trace:
I20260806 20:15:59.600054 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1344) took 1132 ms. Trace:
I20260806 20:15:59.554154 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1396) took 1255 ms. Trace:
W20260806 20:15:59.769732 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 534929us
I20260806 20:15:59.525593 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1342) took 1894 ms. Trace:
W20260806 20:15:59.682852 30706 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.000s sys 0.025s
I20260806 20:15:59.497412 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:59.554047 30792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1689) took 1245 ms. Trace:
I20260806 20:15:59.663323 30676 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:15:59.604384 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:59.551998 30784 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:15:59.559715 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.676025 30757 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:59.507478 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 378495us
I20260806 20:15:59.449534 555 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.847157 388 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:15:59.474694 30865 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1272) took 1411 ms (client timeout 1857 ms). Trace:
I20260806 20:15:59.508957 489 raft_consensus.cc:979] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: : Instructing follower 42cabca48db7403b8523a3382a046043 to start an election
W20260806 20:15:59.835371 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 221401us
I20260806 20:15:59.501864 615 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:15:59.598320 601 log.cc:924] Time spent T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 1.201s user 0.000s sys 0.006s
I20260806 20:15:59.795082 30870 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
W20260806 20:15:59.793258 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 130046us
I20260806 20:15:59.876781 30676 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:15:59.793532 30785 rpcz_store.cc:269] 0806 20:15:57.621697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.621817 (+ 120us) service_pool.cc:227] Handling call
0806 20:15:59.575054 (+1953237us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b5400260f54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.575215 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3292736896}
I20260806 20:15:59.790053 30707 rpcz_store.cc:276] 0806 20:15:58.468138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.574423 (+1106285us) service_pool.cc:227] Handling call
0806 20:15:59.599498 (+ 25075us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:59.599718 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:59.599744 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:15:59.599784 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:59.599815 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:59.599870 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:59.599989 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:15:59.786123 30705 rpcz_store.cc:276] 0806 20:15:57.768120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.768251 (+ 131us) service_pool.cc:227] Handling call
0806 20:15:57.797430 (+ 29179us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:57.815578 (+ 18148us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:15:58.062513 (+246935us) log.cc:842] Serialized 14 byte log entry
0806 20:15:58.325768 (+263255us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:58.326363 (+ 595us) log.cc:842] Serialized 26 byte log entry
0806 20:15:58.326488 (+ 125us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:58.326682 (+ 194us) log.cc:842] Serialized 14 byte log entry
0806 20:15:58.597313 (+270631us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:58.597398 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:59.493980 (+896582us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b48001c0158. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71565167f1 00007f715650d8e7 00007f715650d77a 00007f715660195f 00007f7156601627 00007f7156615b2c 00007f7156615ad8 00007f7156615a58 00007f7156615a13 00007f7156614cbc 00007f7155596be6
0806 20:15:59.521208 (+ 27228us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b24001a22f8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e5abcf4 00007f714e5a9dfb 00007f714e638559 00007f714e6381e5 00007f714e637740 00007f715131e578 00007f715131ecd8 00007f71512f66e8 00007f71512f1aea 00007f71512f0f36
0806 20:15:59.522331 (+ 1123us) raft_consensus.cc:1671] finished
0806 20:15:59.522349 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:59.522471 (+ 122us) inbound_call.cc:173] Queueing success response
0806 20:15:59.549018 (+ 26547us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b180043f900. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e753764 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d 00007f71579ab204 00007f71579ab039 00007f714e735dc6
Metrics: {"mutex_wait_us":110365,"prepare.queue_time_us":449276,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":19872,"raft.queue_time_us":110160,"raft.run_cpu_time_us":8723,"raft.run_wall_time_us":1163275,"spinlock_wait_cycles":186971136,"thread_start_us":679572,"threads_started":3,"wal-append.queue_time_us":397734}
I20260806 20:15:59.585593 614 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:15:59.835845 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 159962us
I20260806 20:15:59.800379 30710 raft_consensus.cc:2379] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:59.882506 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:59.883198 30710 raft_consensus.cc:2379] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
W20260806 20:15:59.794330 30706 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.004s sys 0.027s
W20260806 20:15:59.578745 628 log.cc:924] Time spent T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.197s user 0.000s sys 0.001s
I20260806 20:15:59.792102 30868 rpcz_store.cc:276] 0806 20:15:58.299084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.299315 (+ 231us) service_pool.cc:227] Handling call
0806 20:15:59.553907 (+1254592us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b54002fbcd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.554088 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1024408832}
I20260806 20:15:59.443162 31414 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:15:59.790707 30789 rpcz_store.cc:276] 0806 20:15:57.631137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.673273 (+ 42136us) service_pool.cc:227] Handling call
0806 20:15:59.525334 (+1852061us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400260554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.525511 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3229692672}
W20260806 20:15:59.572788 595 log.cc:924] Time spent T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 1.208s user 0.003s sys 0.000s
W20260806 20:15:59.834607 592 log.cc:924] Time spent T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260806 20:15:59.794098 30712 rpcz_store.cc:276] 0806 20:15:58.559009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.605196 (+ 46187us) service_pool.cc:227] Handling call
0806 20:15:59.569884 (+964688us) spinlock_profiling.cc:243] Waited 53.3 ms on lock 0x7b5400211bd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.570031 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122567040}
I20260806 20:15:59.448382 631 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:15:59.840533 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:59.856217 30711 raft_consensus.cc:2379] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:59.585508 32238 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:59.449683 549 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.624583 543 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:59.504963 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1684) took 1700 ms. Trace:
I20260806 20:15:59.451256 32733 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:59.739079 32757 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:15:59.607182 547 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:15:59.788925 30708 rpcz_store.cc:276] 0806 20:15:58.315916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.316291 (+ 375us) service_pool.cc:227] Handling call
0806 20:15:59.488526 (+1172235us) spinlock_profiling.cc:243] Waited 56.7 ms on lock 0x7b54000c9254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.488720 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":130457344}
I20260806 20:15:59.932089 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:59.933346 30713 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:15:59.839620 425 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:15:59.839475 599 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:15:59.794248 30792 rpcz_store.cc:276] 0806 20:15:58.308741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.308897 (+ 156us) service_pool.cc:227] Handling call
0806 20:15:59.553817 (+1244920us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b5400141ed4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.553977 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12857600}
I20260806 20:15:59.836733 30783 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:59.860639 506 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
W20260806 20:15:59.551113 30790 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53732 (request call id 168) took 1736 ms (client timeout 1604 ms). Trace:
I20260806 20:15:59.916428 631 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:59.961542 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.789865 30787 rpcz_store.cc:276] 0806 20:15:58.324928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.325227 (+ 299us) service_pool.cc:227] Handling call
0806 20:15:59.590606 (+1265379us) spinlock_profiling.cc:243] Waited 80.5 ms on lock 0x7b54003b1ad4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.590745 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185189632}
I20260806 20:15:59.492595 32672 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.487567 32647 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.884727 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1345) took 1270 ms. Trace:
I20260806 20:15:59.865761 388 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:15:59.908411 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:15:59.883963 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:59.969897 30710 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:15:59.969056 30785 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:15:59.909757 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:15:59.795625 514 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.867965 535 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:15:59.912156 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:15:59.913359 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:15:59.914863 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:15:59.915855 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 481170us
I20260806 20:15:59.979737 30789 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:15:59.884951 30861 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.004s sys 0.024s
I20260806 20:15:59.880226 555 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:15:59.910334 339 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.011s sys 0.014s
I20260806 20:15:59.918431 32756 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
W20260806 20:15:59.939899 538 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.008s sys 0.019s
W20260806 20:16:00.026381 538 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s user 0.009s sys 0.019s
I20260806 20:15:59.930243 32238 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.489349 619 consensus_queue.cc:260] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.909521 32507 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.028160 32507 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:15:59.954133 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:15:59.865864 30865 rpcz_store.cc:269] 0806 20:15:58.062808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.128481 (+ 65673us) service_pool.cc:227] Handling call
0806 20:15:59.474495 (+1346014us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b5400375354. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.474605 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70171008}
I20260806 20:15:59.959093 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.960115 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:00.031303 30871 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:00.032289 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:00.032889 30871 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:59.964241 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.965962 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.971489 30791 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:16:00.035215 30711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1341) took 1615 ms (client timeout 2143 ms). Trace:
W20260806 20:16:00.035696 30711 rpcz_store.cc:269] 0806 20:15:58.419678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.419835 (+ 157us) service_pool.cc:227] Handling call
0806 20:16:00.035010 (+1615175us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400174b54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:00.035148 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3188346112}
I20260806 20:15:59.918179 639 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:00.036813 639 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.964673 30706 rpcz_store.cc:276] 0806 20:15:58.614215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.017208 (+402993us) service_pool.cc:227] Handling call
0806 20:15:59.017554 (+ 346us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:59.112152 (+ 94598us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:59.112185 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:59.129006 (+ 16821us) log.cc:842] Serialized 26 byte log entry
0806 20:15:59.884496 (+755490us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:15:59.884525 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:59.884569 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:59.884596 (+ 27us) raft_consensus.cc:1671] finished
0806 20:15:59.884602 (+ 6us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:15:59.884677 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":118532,"raft.queue_time_us":89588,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":6775,"thread_start_us":702768,"threads_started":1,"wal-append.queue_time_us":483225}
I20260806 20:15:59.975606 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.979733 30707 raft_consensus.cc:2379] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:00.038786 30863 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:15:59.989809 30835 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:00.041177 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 51479us
I20260806 20:16:00.041649 568 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:00.042234 568 raft_consensus.cc:686] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:15:59.990324 30868 raft_consensus.cc:1261] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:16:00.031505 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:59.916455 314 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.048s
I20260806 20:16:00.043267 568 consensus_queue.cc:237] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.046476 30865 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:00.025168 555 raft_consensus.cc:686] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:00.052351 555 consensus_queue.cc:237] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.990715 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:15:59.998553 346 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.074345 346 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.002085 586 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.076313 586 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.002595 434 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:00.077780 434 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.037216 30711 raft_consensus.cc:1261] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:16:00.079370 30870 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:00.010636 30834 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:00.081817 616 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:00.012023 596 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:00.083514 596 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.091820 616 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:00.092531 616 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.021217 30601 catalog_manager.cc:5495] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 2 to 3, leader changed from 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 3 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:15:59.557312 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 20:16:00.025404 339 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.651s user 0.011s sys 0.014s
W20260806 20:15:59.940045 400 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.014s sys 0.013s
W20260806 20:16:00.101315 400 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s user 0.014s sys 0.013s
I20260806 20:16:00.031416 30869 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
W20260806 20:15:59.956954 30790 rpcz_store.cc:269] 0806 20:15:57.814895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.815295 (+ 400us) service_pool.cc:227] Handling call
0806 20:15:59.550860 (+1735565us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54003d40d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:15:59.551020 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2680167168}
I20260806 20:16:00.034176 30867 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
W20260806 20:16:00.151970 550 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.015s sys 0.011s
W20260806 20:16:00.154498 31423 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.017s sys 0.008s
I20260806 20:16:00.173776 30705 raft_consensus.cc:2379] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
W20260806 20:16:00.174047 31423 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.550s user 0.017s sys 0.009s
I20260806 20:16:00.099998 30601 catalog_manager.cc:5495] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 4, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 4 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:00.060070 313 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:00.178932 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:15:59.967279 631 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:00.179744 30867 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:15:59.960824 425 raft_consensus.cc:686] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:00.183986 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1393) took 2124 ms. Trace:
W20260806 20:16:00.168010 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 157459us
I20260806 20:16:00.191282 30834 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:00.192730 30834 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:15:59.992112 30861 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s user 0.004s sys 0.024s
I20260806 20:15:59.975666 388 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:00.153398 550 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s user 0.015s sys 0.012s
I20260806 20:15:59.949816 32757 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:00.187510 30864 rpcz_store.cc:276] 0806 20:15:58.059520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.133737 (+ 74217us) service_pool.cc:227] Handling call
0806 20:15:58.134041 (+ 304us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:58.134369 (+ 328us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:58.134395 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:58.161795 (+ 27400us) log.cc:842] Serialized 26 byte log entry
0806 20:15:58.316703 (+154908us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:15:58.849013 (+532310us) log.cc:842] Serialized 14 byte log entry
0806 20:15:58.849295 (+ 282us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:58.849363 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:59.597463 (+748100us) spinlock_profiling.cc:243] Waited 2.76 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:00.172979 (+575516us) raft_consensus.cc:1671] finished
0806 20:16:00.183762 (+ 10783us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:00.183920 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":361738,"prepare.queue_time_us":429140,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":22501,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":11122,"spinlock_wait_cycles":6339840,"thread_start_us":226514,"threads_started":2,"wal-append.queue_time_us":217164}
I20260806 20:16:00.245587 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1532) took 2172 ms. Trace:
I20260806 20:15:59.972162 439 raft_consensus.cc:1067] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Signalling peer 42cabca48db7403b8523a3382a046043 to start an election
W20260806 20:16:00.154273 608 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.015s sys 0.011s
I20260806 20:16:00.184764 31388 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:00.253019 30865 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
W20260806 20:16:00.248198 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 55423.3us
I20260806 20:15:59.990371 32443 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:15:59.950951 391 consensus_queue.cc:1046] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.246s
I20260806 20:16:00.159583 32238 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:15:59.949379 547 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:00.158761 596 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:00.264559 30835 leader_election.cc:398] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3.
I20260806 20:16:00.164317 639 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:15:59.961382 506 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:15:59.939574 543 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:00.268616 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 87168.4us
I20260806 20:16:00.165589 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:00.292852 31417 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:00.165500 616 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:00.296595 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:00.297366 31421 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:16:00.298725 346 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:00.299461 30832 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:00.299885 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:00.300534 346 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:00.300964 346 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:00.301029 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:00.284600 30602 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41560 (request call id 217) took 6328 ms. Trace:
I20260806 20:16:00.312029 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 172) took 1019 ms. Trace:
I20260806 20:16:00.310935 30790 raft_consensus.cc:2362] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5. Re-sending same reply.
I20260806 20:16:00.312440 30788 rpcz_store.cc:276] 0806 20:15:59.292350 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.438435 (+146085us) service_pool.cc:227] Handling call
0806 20:15:59.438732 (+ 297us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:59.439066 (+ 334us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:59.439092 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:59.599648 (+160556us) spinlock_profiling.cc:243] Waited 4.2 ms on lock 0x7b2c000002e8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:15:59.600063 (+ 415us) log.cc:842] Serialized 26 byte log entry
0806 20:16:00.049961 (+449898us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:00.050097 (+ 136us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:00.050438 (+ 341us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:00.311703 (+261265us) raft_consensus.cc:1671] finished
0806 20:16:00.311723 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:00.311970 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":354,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":90,"spinlock_wait_cycles":9654016,"thread_start_us":448616,"threads_started":1,"wal-append.queue_time_us":692436}
I20260806 20:16:00.315580 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:00.316972 30867 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
W20260806 20:16:00.267175 608 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.815s user 0.016s sys 0.012s
I20260806 20:16:00.264029 30703 rpcz_store.cc:276] 0806 20:15:58.073013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:58.073165 (+ 152us) service_pool.cc:227] Handling call
0806 20:15:58.073524 (+ 359us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:58.073834 (+ 310us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:58.073862 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:58.074310 (+ 448us) log.cc:842] Serialized 26 byte log entry
0806 20:15:58.276691 (+202381us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:15:58.276905 (+ 214us) log.cc:842] Serialized 14 byte log entry
0806 20:15:58.779153 (+502248us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b180043f9c0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f71512fe82c 00007f714e7359dc 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e750155 00007f71516673d0 00007f7151679576 00007f715168a0d7
0806 20:15:58.779294 (+ 141us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:58.779367 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:15:59.572158 (+792791us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:00.245338 (+673180us) raft_consensus.cc:1671] finished
0806 20:16:00.245362 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:00.245524 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6855,"prepare.queue_time_us":38624,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":2533,"raft.queue_time_us":83032,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":30242,"spinlock_wait_cycles":19425280,"thread_start_us":247712,"threads_started":2,"wal-append.queue_time_us":243437}
W20260806 20:16:00.256860 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 374417us
I20260806 20:16:00.290427 434 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:00.264338 640 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260806 20:16:00.291432 586 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:00.291985 32507 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:00.301388 346 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:00.306998 667 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:00.308557 351 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:00.310160 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1537) took 1593 ms. Trace:
I20260806 20:16:00.174458 535 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.248598 31420 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:00.265056 566 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:00.273066 30835 leader_election.cc:401] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:15:59.918561 30786 rpcz_store.cc:276] 0806 20:15:57.804806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:57.805226 (+ 420us) service_pool.cc:227] Handling call
0806 20:15:57.841695 (+ 36469us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:58.759312 (+917617us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:59.504612 (+745300us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b54001d0bd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:15:59.504859 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2108885888}
I20260806 20:16:00.248665 567 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:00.259075 578 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:00.314136 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:00.195689 313 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.326448 30710 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:16:00.329391 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1362) took 1002 ms. Trace:
I20260806 20:16:00.331017 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:00.891345 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 488833us
W20260806 20:16:00.865948 30676 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.001s sys 0.006s
W20260806 20:16:00.781641 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 470924us
W20260806 20:16:00.894767 30676 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.568s user 0.001s sys 0.007s
W20260806 20:16:00.837312 30787 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.001s sys 0.008s
W20260806 20:16:00.896247 30787 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.627s user 0.002s sys 0.009s
I20260806 20:16:00.890493 30705 rpcz_store.cc:276] 0806 20:15:59.326671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.878729 (+552058us) service_pool.cc:227] Handling call
0806 20:16:00.329182 (+450453us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54000c7454. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:00.329325 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":446098688}
I20260806 20:16:00.390863 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:00.353296 31415 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:16:00.892674 30834 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:00.895421 30709 rpcz_store.cc:276] 0806 20:15:58.716873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.210319 (+493446us) service_pool.cc:227] Handling call
0806 20:15:59.210678 (+ 359us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:15:59.310859 (+100181us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:59.310890 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:15:59.311467 (+ 577us) log.cc:842] Serialized 52 byte log entry
0806 20:15:59.477609 (+166142us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b180043f720. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e735345 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb 00007f71512f6063 00007f71512f6411 00007f71516435cf
0806 20:15:59.931018 (+453409us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:15:59.952335 (+ 21317us) log.cc:842] Serialized 14 byte log entry
0806 20:15:59.952762 (+ 427us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:15:59.952852 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:00.309868 (+357016us) raft_consensus.cc:1671] finished
0806 20:16:00.309973 (+ 105us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:00.310098 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34107,"prepare.queue_time_us":20646,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":32292,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":31789,"spinlock_wait_cycles":6166400,"thread_start_us":453184,"threads_started":1,"wal-append.queue_time_us":977720}
I20260806 20:16:00.903746 30703 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:16:00.389974 30601 catalog_manager.cc:5495] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 5, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 5 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:00.267737 634 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:00.905494 634 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:00.910419 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 649383us
I20260806 20:16:00.912083 30835 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:16:00.895184 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 568674us
I20260806 20:16:00.278786 400 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:00.574437 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:00.621891 30712 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.023s sys 0.022s
W20260806 20:16:00.917490 30712 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.966s user 0.026s sys 0.023s
W20260806 20:16:00.496940 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 44571.4us
I20260806 20:16:00.918563 30673 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:16:00.354198 601 log.cc:924] Time spent T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260806 20:16:00.310915 30602 rpcz_store.cc:276] 0806 20:15:53.934056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:53.934407 (+ 351us) service_pool.cc:227] Handling call
0806 20:16:00.262675 (+6328268us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:15:57.390600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:15:57.415029 (+ 24429us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:15:57.415077 (+ 48us) write_op.cc:435] Acquired schema lock
0806 20:15:57.415094 (+ 17us) tablet.cc:660] Decoding operations
0806 20:15:57.526282 (+111188us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:15:57.526406 (+ 124us) write_op.cc:641] Partition lock acquired for write op
0806 20:15:57.526449 (+ 43us) tablet.cc:683] Acquiring locks for 24 operations
0806 20:15:57.666619 (+140170us) tablet.cc:699] Row locks acquired
0806 20:15:57.666658 (+ 39us) write_op.cc:260] PREPARE: finished
0806 20:15:57.666865 (+ 207us) write_op.cc:270] Start()
0806 20:15:57.667049 (+ 184us) write_op.cc:276] Timestamp: P: 1786047357666776 usec, L: 0
0806 20:15:57.667090 (+ 41us) op_driver.cc:331] REPLICATION: starting
0806 20:15:57.812360 (+145270us) log.cc:842] Serialized 9573 byte log entry
0806 20:15:59.099900 (+1287540us) op_driver.cc:444] REPLICATION: finished
0806 20:15:59.629641 (+529741us) write_op.cc:301] APPLY: starting
0806 20:15:59.630363 (+ 722us) tablet.cc:1365] starting BulkCheckPresence
0806 20:15:59.633660 (+ 3297us) tablet.cc:1368] finished BulkCheckPresence
0806 20:15:59.633829 (+ 169us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:15:59.642477 (+ 8648us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:15:59.642857 (+ 380us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=24
0806 20:15:59.642900 (+ 43us) write_op.cc:312] APPLY: finished
0806 20:15:59.643414 (+ 514us) log.cc:842] Serialized 164 byte log entry
0806 20:15:59.738784 (+ 95370us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:59.739045 (+ 261us) write_op.cc:454] Released schema lock
0806 20:15:59.739140 (+ 95us) write_op.cc:341] FINISH: Updating metrics
0806 20:15:59.739490 (+ 350us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:15:59.739521 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":529021,"apply.run_cpu_time_us":6472,"apply.run_wall_time_us":110997,"num_ops":24,"prepare.queue_time_us":207524,"prepare.run_cpu_time_us":19362,"prepare.run_wall_time_us":822617,"raft.queue_time_us":343328,"raft.run_cpu_time_us":57384,"raft.run_wall_time_us":311580,"replication_time_us":1430564,"thread_start_us":795679,"threads_started":4,"wal-append.queue_time_us":538230}]]}
I20260806 20:16:00.328931 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:00.331362 613 consensus_meta.cc:287] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.920s user 0.009s sys 0.004s
W20260806 20:16:00.824088 31416 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.016s sys 0.017s
W20260806 20:16:00.726996 661 log.cc:924] Time spent T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
W20260806 20:16:00.813537 568 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s user 0.012s sys 0.013s
I20260806 20:16:00.801159 32507 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.800846 605 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.782492 32672 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:00.386399 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1399) took 1248 ms. Trace:
I20260806 20:16:00.496515 615 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:00.829241 32647 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:00.510004 32733 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:00.407769 31417 raft_consensus.cc:1226] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:00.529019 659 consensus_queue.cc:1035] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.487s
I20260806 20:16:00.922529 566 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.907768 618 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:00.827098 608 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.962044 555 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:16:00.385607 655 log.cc:924] Time spent T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.091s user 0.000s sys 0.006s
I20260806 20:16:01.002347 543 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:00.774976 586 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:00.280972 32443 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:00.962210 31416 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.889s user 0.028s sys 0.024s
I20260806 20:16:00.843657 599 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.992324 32507 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:00.528460 637 log.cc:924] Time spent T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
I20260806 20:16:00.961627 387 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.961824 578 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:00.969175 564 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:01.018913 660 log.cc:924] Time spent T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.253s user 0.000s sys 0.001s
I20260806 20:16:01.001804 605 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.986755 30711 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:16:00.445495 425 consensus_queue.cc:237] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.000399 549 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:00.325165 597 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:00.340370 667 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:00.927791 351 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.024135 30601 catalog_manager.cc:5495] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:01.037784 618 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.035620 514 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:01.047497 30789 consensus_meta.cc:287] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.055s user 0.008s sys 0.000s
I20260806 20:16:01.097621 550 raft_consensus.cc:3032] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:01.021361 506 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.035110 439 raft_consensus.cc:3037] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:16:01.038522 548 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:01.129146 400 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:16:01.186307 30788 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.009s sys 0.031s
I20260806 20:16:00.911401 567 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:01.189940 31423 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.885s user 0.020s sys 0.047s
I20260806 20:16:01.031612 547 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.002624 613 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.038656 608 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.196122 550 raft_consensus.cc:729] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:01.189939 30601 catalog_manager.cc:5495] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 3, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 3 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:01.216565 548 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:01.190135 30789 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:01.234169 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1364) took 1365 ms. Trace:
I20260806 20:16:01.234505 30789 rpcz_store.cc:276] 0806 20:15:59.868371 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.914609 (+ 46238us) service_pool.cc:227] Handling call
0806 20:16:01.234098 (+1319489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:01.235548 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:01.236706 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.197480 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:01.198644 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:01.237787 30865 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:01.200094 30864 raft_consensus.cc:1261] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:16:01.021100 32733 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.017144 568 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:16:01.045667 543 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:01.244807 543 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:01.246850 677 log.cc:924] Time spent T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
I20260806 20:16:01.247135 543 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.232900 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.186592 549 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.253132 549 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.241092 548 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:01.177912 664 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:16:01.216198 30788 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.896s user 0.010s sys 0.031s
I20260806 20:16:01.256609 30601 catalog_manager.cc:5495] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:01.179979 599 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:01.187903 669 log.cc:924] Time spent T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.171s user 0.000s sys 0.001s
I20260806 20:16:01.183405 535 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:01.220031 31423 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.915s user 0.020s sys 0.047s
I20260806 20:16:01.185986 634 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.031491 566 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.073889 32757 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.041126 555 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.221525 400 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:01.213045 662 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.894s user 0.019s sys 0.041s
I20260806 20:16:01.043303 586 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:00.997603 30861 rpcz_store.cc:276] 0806 20:15:59.103818 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.121440 (+ 17622us) service_pool.cc:227] Handling call
0806 20:15:59.204229 (+ 82789us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:59.231380 (+ 27151us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 20:15:59.289913 (+ 58533us) log.cc:842] Serialized 14 byte log entry
0806 20:16:00.268520 (+978607us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:00.270892 (+ 2372us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:00.286281 (+ 15389us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:00.324388 (+ 38107us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:00.351812 (+ 27424us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":41835,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":2747,"raft.queue_time_us":4798,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11660,"spinlock_wait_cycles":46517376,"thread_start_us":956353,"threads_started":1,"wal-append.queue_time_us":690240}
I20260806 20:16:01.292865 30867 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:01.231985 32507 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:01.179476 564 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.179673 387 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.179817 578 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.243901 351 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.244004 568 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:16:01.292889 662 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.974s user 0.019s sys 0.042s
I20260806 20:16:01.264727 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:01.274389 30790 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:16:01.320338 30789 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:01.290771 667 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.278864 567 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:01.279570 31418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48918 (request call id 362) took 2007 ms. Trace:
I20260806 20:16:01.280401 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1711) took 1413 ms. Trace:
I20260806 20:16:01.040769 32672 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:01.046592 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:01.285977 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 341027us
W20260806 20:16:01.274201 631 consensus_meta.cc:287] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.024s user 0.000s sys 0.011s
I20260806 20:16:01.283471 535 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:16:01.230419 388 consensus_meta.cc:287] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.962s user 0.000s sys 0.012s
W20260806 20:16:01.340835 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 837101us
I20260806 20:16:01.291813 613 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.314823 387 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.316656 578 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.288398 400 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.317832 568 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:01.288475 566 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:01.330029 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1358) took 2004 ms. Trace:
I20260806 20:16:01.293403 634 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:01.313200 564 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:01.356274 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.360816 31388 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:01.275605 30865 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:01.309288 32574 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:01.271248 664 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.334084 30791 rpcz_store.cc:276] 0806 20:15:59.867189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.034878 (+167689us) service_pool.cc:227] Handling call
0806 20:16:00.035239 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:00.035577 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:00.035603 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:00.036080 (+ 477us) log.cc:842] Serialized 26 byte log entry
0806 20:16:00.385580 (+349500us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:00.385619 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:00.385688 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:01.280156 (+894468us) raft_consensus.cc:1671] finished
0806 20:16:01.280178 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.280334 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8823,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":286870,"threads_started":1,"wal-append.queue_time_us":866068}
I20260806 20:16:01.270298 313 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:01.350244 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 431754us
I20260806 20:16:01.351621 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.351840 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:01.368739 31421 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:16:01.369050 30711 raft_consensus.cc:2396] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 20:16:01.340469 605 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:01.338363 535 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:01.293956 550 consensus_queue.cc:260] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.345775 549 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:01.352698 30787 rpcz_store.cc:276] 0806 20:15:59.325730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.969235 (+643505us) service_pool.cc:227] Handling call
0806 20:16:00.172792 (+203557us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:00.173024 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:16:00.896723 (+723699us) log.cc:842] Serialized 14 byte log entry
0806 20:16:01.329491 (+432768us) log.cc:842] Serialized 14 byte log entry
0806 20:16:01.329659 (+ 168us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:01.329718 (+ 59us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:01.329771 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.329841 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.329964 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73084,"prepare.queue_time_us":100188,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":4658,"raft.queue_time_us":2390,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":3124,"spinlock_wait_cycles":17449216,"thread_start_us":1059149,"threads_started":2}
W20260806 20:16:01.333277 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 421254us
I20260806 20:16:01.337286 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1545) took 1471 ms. Trace:
I20260806 20:16:01.369867 30832 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:01.398160 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.374939 564 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:01.377369 535 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:01.350693 32443 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.374083 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.374775 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.340623 631 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.339939 32672 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.381671 30789 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:01.382081 640 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:01.385514 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1718) took 1091 ms. Trace:
I20260806 20:16:01.399554 524 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:01.359946 634 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:16:01.423837 30788 rpcz_store.cc:276] 0806 20:16:00.294095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.316500 (+ 22405us) service_pool.cc:227] Handling call
0806 20:16:00.316937 (+ 437us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:00.317144 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:00.317174 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:00.320113 (+ 2939us) log.cc:842] Serialized 26 byte log entry
0806 20:16:01.256786 (+936673us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:16:01.256825 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.256888 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:01.384499 (+127611us) spinlock_profiling.cc:243] Waited 4.85 ms on lock 0x7b24001051b8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e5abcf4 00007f714e5a9dfb 00007f714e638559 00007f714e6381e5 00007f714e637740 00007f715131e578 00007f715131ecd8 00007f71512f66e8 00007f71512f1aea 00007f71512f0f36
0806 20:16:01.385195 (+ 696us) raft_consensus.cc:1671] finished
0806 20:16:01.385216 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.385429 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11161472,"thread_start_us":936314,"threads_started":1,"wal-append.queue_time_us":956124}
I20260806 20:16:01.392838 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 174) took 1098 ms. Trace:
W20260806 20:16:01.393718 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 410123us
I20260806 20:16:01.357502 689 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:01.418720 547 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:01.435519 689 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.435701 547 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.437723 689 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.440840 619 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:01.444595 346 consensus_meta.cc:287] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.092s user 0.008s sys 0.004s
I20260806 20:16:01.445858 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.411557 564 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.446722 31421 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:01.448153 564 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.449038 618 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.423884 640 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.451615 640 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.424322 524 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.454193 524 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.454376 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:01.460052 31414 consensus_meta.cc:287] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.542s user 0.011s sys 0.003s
I20260806 20:16:01.460558 31414 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
W20260806 20:16:01.461187 31414 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49046 (request call id 529) took 2110 ms (client timeout 2070 ms). Trace:
W20260806 20:16:01.461560 31414 rpcz_store.cc:269] 0806 20:15:59.350334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.357590 (+ 7256us) service_pool.cc:227] Handling call
0806 20:16:01.461069 (+2103479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:01.465694 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.347980 388 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.370862 351 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:01.398360 30835 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:01.474882 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 76606.2us
I20260806 20:16:01.370827 31388 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:01.411648 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1361) took 2098 ms. Trace:
I20260806 20:16:01.400046 32733 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.364413 32647 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 20:16:01.414827 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:01.398733 30707 rpcz_store.cc:276] 0806 20:15:59.866206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.039056 (+172850us) service_pool.cc:227] Handling call
0806 20:16:00.039374 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:00.039704 (+ 330us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:16:00.039980 (+ 276us) log.cc:842] Serialized 14 byte log entry
0806 20:16:00.521800 (+481820us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:00.718935 (+197135us) log.cc:842] Serialized 26 byte log entry
0806 20:16:00.893270 (+174335us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:01.285593 (+392323us) log.cc:842] Serialized 14 byte log entry
0806 20:16:01.336958 (+ 51365us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.337041 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:01.337082 (+ 41us) raft_consensus.cc:1671] finished
0806 20:16:01.337092 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.337225 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":358638,"prepare.queue_time_us":815,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":132563,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":76120,"thread_start_us":796130,"threads_started":2,"wal-append.queue_time_us":1118367}
I20260806 20:16:01.333915 31418 rpcz_store.cc:276] 0806 20:15:59.266663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.273207 (+ 6544us) service_pool.cc:227] Handling call
0806 20:15:59.359020 (+ 85813us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:15:59.624959 (+265939us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:15:59.624998 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:15:59.625665 (+ 667us) log.cc:842] Serialized 26 byte log entry
0806 20:16:00.172855 (+547190us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:00.179348 (+ 6493us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:00.195490 (+ 16142us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:01.260290 (+1064800us) raft_consensus.cc:1671] finished
0806 20:16:01.260312 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.273719 (+ 13407us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":239104,"raft.queue_time_us":241500,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":338792,"threads_started":1,"wal-append.queue_time_us":815336}
I20260806 20:16:01.417531 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.418148 663 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:01.393281 549 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.400269 550 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:01.401641 605 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:01.377030 614 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:01.401697 599 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.399935 608 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.372918 32574 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.401070 578 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:01.403752 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.432904 30673 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:01.434209 30783 rpcz_store.cc:276] 0806 20:16:00.294356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.401721 (+107365us) service_pool.cc:227] Handling call
0806 20:16:00.751561 (+349840us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:01.286530 (+534969us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:01.293122 (+ 6592us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:01.306603 (+ 13481us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:01.308605 (+ 2002us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.392576 (+ 83971us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x7b54003b15d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:01.392627 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.392775 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1199044352}
I20260806 20:16:01.417591 555 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:01.476382 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 105631us
I20260806 20:16:01.526404 605 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:01.528970 614 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:01.536978 614 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:01.445119 346 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.539134 346 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.475955 30712 rpcz_store.cc:276] 0806 20:15:59.313315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.937780 (+624465us) service_pool.cc:227] Handling call
0806 20:15:59.938039 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:15:59.938239 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:15:59.951437 (+ 13198us) log.cc:842] Serialized 14 byte log entry
0806 20:16:01.411009 (+1459572us) log.cc:842] Serialized 14 byte log entry
0806 20:16:01.411295 (+ 286us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:01.411346 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:01.411395 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.411467 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.411590 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":81018,"prepare.queue_time_us":485331,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":65,"raft.queue_time_us":109958,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":14949,"thread_start_us":1386129,"threads_started":2,"wal-append.queue_time_us":1274190}
I20260806 20:16:01.541105 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.541790 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.542418 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.542919 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.543695 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:01.544081 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.544293 30707 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:16:01.545059 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:01.530856 32574 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:01.511773 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 97004.3us
I20260806 20:16:01.494050 640 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:01.546960 30676 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:01.547978 685 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:01.548530 605 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:01.497793 564 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:01.549681 564 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:01.550060 564 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:01.497058 351 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.542994 31419 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:01.533286 578 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.535203 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:01.535084 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 102270us
I20260806 20:16:01.471689 664 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.496524 586 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:01.566962 31420 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 433) took 2461 ms. Trace:
I20260806 20:16:01.576749 613 consensus_queue.cc:1046] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260806 20:16:01.591995 439 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 2. Current term is 3.
I20260806 20:16:01.593765 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.524730 32733 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:01.597611 31416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48918 (request call id 366) took 1525 ms. Trace:
I20260806 20:16:01.599478 640 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:01.600098 640 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.516561 547 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.548485 685 raft_consensus.cc:686] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:01.572402 30835 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:01.567682 346 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:01.582206 31420 rpcz_store.cc:276] 0806 20:15:59.105157 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.121182 (+ 16025us) service_pool.cc:227] Handling call
0806 20:15:59.146337 (+ 25155us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:00.900779 (+1754442us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.566676 (+665897us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54002cecd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:01.566865 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2366194048}
W20260806 20:16:01.622210 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 49908.7us
I20260806 20:16:01.611932 30792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 176) took 1317 ms. Trace:
I20260806 20:16:01.622934 30792 rpcz_store.cc:276] 0806 20:16:00.294896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.394246 (+ 99350us) service_pool.cc:227] Handling call
0806 20:16:00.497933 (+103687us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:01.215903 (+717970us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:01.215956 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:01.215998 (+ 42us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:01.216026 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.611679 (+395653us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b54003b24d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:01.611727 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.611870 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1650733824}
I20260806 20:16:01.632263 32443 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:01.535422 555 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:01.534442 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.561254 689 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:01.511296 663 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.524119 566 consensus_queue.cc:1046] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20260806 20:16:01.548496 30676 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:01.600847 535 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.602864 32733 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:01.632153 346 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.825758 30705 raft_consensus.cc:1261] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:01.826587 32443 raft_consensus.cc:3032] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:01.826990 32443 raft_consensus.cc:729] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:01.808249 631 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.692818 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.830163 32507 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:01.788303 31423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.006s sys 0.022s
W20260806 20:16:01.682020 30834 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.778s user 0.000s sys 0.023s
W20260806 20:16:01.821929 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 257252us
W20260806 20:16:01.854694 31423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s user 0.006s sys 0.022s
W20260806 20:16:01.854899 30834 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.951s user 0.003s sys 0.027s
W20260806 20:16:01.857328 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 964678us
I20260806 20:16:01.858134 30834 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:01.848701 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 188769us
I20260806 20:16:01.827883 663 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.825629 32733 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:01.836988 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 177) took 1541 ms. Trace:
I20260806 20:16:01.838778 689 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.644160 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 206) took 2283 ms. Trace:
W20260806 20:16:01.843247 30601 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.025s sys 0.034s
I20260806 20:16:01.632128 613 raft_consensus.cc:3032] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:01.641196 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.867580 30790 rpcz_store.cc:276] 0806 20:16:00.295250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.905906 (+610656us) service_pool.cc:227] Handling call
0806 20:16:00.906353 (+ 447us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:01.320222 (+413869us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.836720 (+516498us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b54003d45d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:01.836923 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":845622016}
I20260806 20:16:01.878497 30710 rpcz_store.cc:276] 0806 20:15:59.360927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:59.970585 (+609658us) service_pool.cc:227] Handling call
0806 20:15:59.970917 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:00.889934 (+919017us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.643884 (+753950us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b54001b4254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:01.644089 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":816810368}
W20260806 20:16:01.878633 30601 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.550s user 0.025s sys 0.034s
W20260806 20:16:01.866909 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 318462us
I20260806 20:16:01.885613 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.888849 30792 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:01.884276 31423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 531) took 1596 ms. Trace:
I20260806 20:16:01.887053 30676 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:01.895442 30676 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:01.897923 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:01.846558 662 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.024s sys 0.034s
I20260806 20:16:01.826066 30786 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:16:01.846074 547 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:01.890381 31423 rpcz_store.cc:276] 0806 20:16:00.287529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.288190 (+ 661us) service_pool.cc:227] Handling call
0806 20:16:00.288517 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:00.288849 (+ 332us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 20:16:01.268667 (+979818us) log.cc:842] Serialized 14 byte log entry
0806 20:16:01.883883 (+615216us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:01.883944 (+ 61us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:01.884000 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.884074 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.884212 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7193,"prepare.queue_time_us":806408,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":27203,"thread_start_us":1551710,"threads_started":2}
I20260806 20:16:01.911087 31389 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:01.846891 640 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.886802 613 raft_consensus.cc:729] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:01.886106 618 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:01.921336 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.929350 30790 raft_consensus.cc:2396] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260806 20:16:01.915195 662 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s user 0.025s sys 0.034s
I20260806 20:16:01.878959 32672 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.937132 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:01.922227 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:01.920925 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.827872 32443 consensus_queue.cc:260] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.930976 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:01.912101 608 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:01.865928 32507 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.866114 32574 consensus_queue.cc:1046] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260806 20:16:01.934520 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:01.926549 690 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:01.885914 631 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:01.966445 30756 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:01.967716 30756 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:01.968016 550 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:01.968386 32642 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:01.968688 550 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.970000 313 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:01.971050 550 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:01.971477 645 log.cc:924] Time spent T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 1.708s user 0.006s sys 0.014s
I20260806 20:16:01.939785 30835 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:01.971717 631 raft_consensus.cc:686] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:01.972822 631 consensus_queue.cc:237] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.973759 578 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.957423 608 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.975795 614 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.976922 31422 raft_consensus.cc:1226] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
I20260806 20:16:01.977995 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:01.934885 618 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:01.979212 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:01.863615 555 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:01.906247 564 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.993048 31415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 438) took 1945 ms. Trace:
I20260806 20:16:01.948778 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.950461 689 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:01.913182 346 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:01.950773 535 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:01.986243 32642 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.911734 695 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260806 20:16:01.951081 388 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.878160 685 consensus_queue.cc:237] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:01.951871 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.003598 32642 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:02.004094 30707 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 1.
I20260806 20:16:02.005019 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.005847 30707 raft_consensus.cc:2379] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:02.006160 659 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:16:01.938241 30710 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:02.007894 30710 raft_consensus.cc:1261] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:16:01.917191 634 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:02.009336 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:01.906991 663 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:01.906497 506 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:16:01.932497 548 consensus_meta.cc:287] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.675s user 0.011s sys 0.000s
I20260806 20:16:01.955726 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:02.014853 30790 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:01.993425 31415 rpcz_store.cc:276] 0806 20:16:00.047339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.047490 (+ 151us) service_pool.cc:227] Handling call
0806 20:16:00.047777 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:00.901535 (+853758us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.992796 (+1091261us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b540067f2d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:01.992987 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":701724416}
I20260806 20:16:01.920385 599 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:01.935312 640 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:01.959105 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:01.959164 682 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:16:01.598061 31416 rpcz_store.cc:276] 0806 20:16:00.072569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.072719 (+ 150us) service_pool.cc:227] Handling call
0806 20:16:00.073080 (+ 361us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:00.073327 (+ 247us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:16:01.082193 (+1008866us) log.cc:842] Serialized 14 byte log entry
0806 20:16:01.576279 (+494086us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:01.597205 (+ 20926us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:01.597285 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.597369 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:01.597541 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5462,"prepare.queue_time_us":1002904,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"raft.queue_time_us":60031,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":180,"thread_start_us":1368156,"threads_started":2}
I20260806 20:16:01.978276 30756 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:01.992877 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.977926 32574 raft_consensus.cc:3032] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:01.956126 549 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:01.969354 667 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:16:01.931630 547 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.001065 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:02.010157 30786 raft_consensus.cc:2354] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7.
W20260806 20:16:02.011283 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 418252us
I20260806 20:16:02.012100 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1713) took 1719 ms. Trace:
I20260806 20:16:02.010036 634 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:02.017885 30708 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
W20260806 20:16:01.996429 568 consensus_meta.cc:287] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.635s user 0.010s sys 0.001s
I20260806 20:16:01.959048 613 consensus_queue.cc:260] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:02.028679 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 50505.2us
I20260806 20:16:01.986470 555 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.005254 564 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:01.921483 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:01.940690 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:02.031255 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:02.036600 31417 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.042s sys 0.080s
I20260806 20:16:02.116652 30792 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
W20260806 20:16:02.117079 31417 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.042s sys 0.090s
I20260806 20:16:02.072041 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:16:02.041057 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1417) took 1842 ms. Trace:
I20260806 20:16:02.156697 30869 rpcz_store.cc:276] 0806 20:16:00.199108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.246806 (+ 47698us) service_pool.cc:227] Handling call
0806 20:16:00.247140 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:01.185423 (+938283us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:01.185540 (+ 117us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:01.185580 (+ 40us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:01.185612 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:02.040828 (+855216us) spinlock_profiling.cc:243] Waited 938 ms on lock 0x7b5400358254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:02.040874 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:02.040992 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2157403520}
I20260806 20:16:02.010830 506 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:02.121330 30756 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:02.129751 30789 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.001s sys 0.027s
W20260806 20:16:02.129660 30787 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s user 0.011s sys 0.027s
W20260806 20:16:02.161779 30787 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s user 0.013s sys 0.028s
I20260806 20:16:02.164242 30756 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:02.115988 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 179847us
W20260806 20:16:02.129829 30832 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.007s sys 0.013s
I20260806 20:16:02.164906 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:02.048771 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:02.118073 30835 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:02.165843 30792 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
W20260806 20:16:02.166052 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 48047.3us
I20260806 20:16:02.152029 30785 rpcz_store.cc:276] 0806 20:16:00.292563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.328627 (+ 36064us) service_pool.cc:227] Handling call
0806 20:16:00.328937 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:00.982455 (+653518us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:16:00.982792 (+ 337us) log.cc:842] Serialized 14 byte log entry
0806 20:16:01.344276 (+361484us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:01.346142 (+ 1866us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:01.346213 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:02.011859 (+665646us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b5400012cd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:02.011907 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:02.012039 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":988,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":3467,"raft.queue_time_us":76,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":3118,"spinlock_wait_cycles":1502396288,"thread_start_us":361079,"threads_started":1,"wal-append.queue_time_us":358944}
I20260806 20:16:02.161988 30706 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:02.014375 346 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:02.163956 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 180) took 1843 ms. Trace:
W20260806 20:16:02.129320 30713 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.013s sys 0.025s
W20260806 20:16:02.165313 30832 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.008s sys 0.014s
W20260806 20:16:02.150878 567 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.008s sys 0.012s
I20260806 20:16:02.166795 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:02.117018 663 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:16:02.152555 425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.023s sys 0.013s
W20260806 20:16:02.224169 30713 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.828s user 0.014s sys 0.026s
I20260806 20:16:02.212215 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.224673 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:02.224758 30835 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:02.225100 30783 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 6
W20260806 20:16:02.161306 30789 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s user 0.001s sys 0.029s
I20260806 20:16:02.179592 30676 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:02.226146 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 699293us
W20260806 20:16:02.264281 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 84780.4us
I20260806 20:16:02.223767 30787 rpcz_store.cc:276] 0806 20:16:00.320262 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.389623 (+1069361us) service_pool.cc:227] Handling call
0806 20:16:01.389980 (+ 357us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:01.390190 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:16:01.390535 (+ 345us) log.cc:842] Serialized 14 byte log entry
0806 20:16:02.163618 (+773083us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:02.163671 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:02.163729 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:02.163793 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:02.163901 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":557,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":4180,"thread_start_us":772746,"threads_started":1,"wal-append.queue_time_us":725479}
I20260806 20:16:01.999279 351 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:02.228114 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 46472.9us
W20260806 20:16:02.228206 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 65586.1us
I20260806 20:16:02.267014 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:02.019313 640 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:02.097688 608 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:02.077013 31416 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:02.029754 385 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:02.115650 634 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:02.117308 618 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:02.265367 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:02.269133 425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s user 0.024s sys 0.014s
W20260806 20:16:02.267190 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 356169us
I20260806 20:16:02.151762 32574 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:02.056834 32507 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:02.273460 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 105779us
I20260806 20:16:02.261085 30708 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:02.262974 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:02.273030 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 181) took 1349 ms. Trace:
I20260806 20:16:02.029254 682 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:02.228551 564 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:02.255427 549 consensus_queue.cc:1035] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20260806 20:16:02.331773 30791 rpcz_store.cc:276] 0806 20:16:00.923304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.405238 (+481934us) service_pool.cc:227] Handling call
0806 20:16:01.426101 (+ 20863us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:01.426429 (+ 328us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:01.426453 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:01.426880 (+ 427us) log.cc:842] Serialized 26 byte log entry
0806 20:16:01.955769 (+528889us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:01.955816 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:01.955889 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:02.272838 (+316949us) raft_consensus.cc:1671] finished
0806 20:16:02.272858 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:02.272976 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5776,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":34988,"thread_start_us":453241,"threads_started":1,"wal-append.queue_time_us":604958}
I20260806 20:16:02.164276 506 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:02.243052 313 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:02.177903 32642 raft_consensus.cc:2781] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:16:02.345431 30790 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:16:02.019212 599 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.227715 692 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:02.242659 621 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:02.259234 567 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.730s user 0.013s sys 0.012s
I20260806 20:16:02.254819 555 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:02.017539 548 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:02.162904 518 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:02.245294 614 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:02.268399 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:02.362792 31415 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:16:02.259014 613 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:02.324020 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 58732.1us
I20260806 20:16:02.329753 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:02.340137 564 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:16:02.314399 699 log.cc:924] Time spent T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.244s user 0.000s sys 0.001s
W20260806 20:16:02.336871 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 476777us
I20260806 20:16:02.260767 663 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:02.339314 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:02.317836 689 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:02.160290 568 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.351593 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:02.373404 30832 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:02.319541 634 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:02.257413 346 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.347126 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:02.347898 31422 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:16:02.347836 30709 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:02.278714 640 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:02.242882 691 consensus_queue.cc:1046] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260806 20:16:02.168000 538 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.168s
I20260806 20:16:02.277315 590 consensus_queue.cc:1035] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260806 20:16:02.369046 564 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:02.394373 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:02.352316 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 80258.4us
I20260806 20:16:02.339881 682 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:02.398409 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.399345 30710 raft_consensus.cc:2379] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:02.360253 30835 leader_election.cc:398] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7.
I20260806 20:16:02.330227 694 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:02.337225 618 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:02.271629 690 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:02.365969 30676 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:02.326310 667 consensus_queue.cc:1035] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.025s
I20260806 20:16:02.347393 32642 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:02.364876 30754 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:02.347659 313 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:16:02.434489 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 68589.3us
I20260806 20:16:02.372711 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.387290 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.351640 30791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:02.395731 31388 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:02.395970 30756 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:02.399906 30835 leader_election.cc:401] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260806 20:16:02.407795 597 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:02.410280 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:02.411904 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:02.415616 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.417289 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.419716 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.373378 663 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:02.373301 689 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:02.423982 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:02.421175 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.427534 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.357983 621 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.431442 32642 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:02.385907 32507 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:02.366554 518 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.391402 351 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:02.372211 578 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.056s
I20260806 20:16:02.356757 692 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.370062 32574 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:02.464566 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.717041 30867 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:02.715159 30864 raft_consensus.cc:2379] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:02.721712 30868 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:16:02.464936 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.723917 30866 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:16:02.725081 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:02.725778 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 330180us
I20260806 20:16:02.727620 30870 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:02.729365 30710 raft_consensus.cc:2396] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 20:16:02.715215 30791 raft_consensus.cc:595] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:02.553013 709 log.cc:924] Time spent T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.291s user 0.001s sys 0.000s
W20260806 20:16:02.435426 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 70632.3us
W20260806 20:16:02.716553 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 320670us
I20260806 20:16:02.734654 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.724406 30867 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
W20260806 20:16:02.535652 32733 consensus_meta.cc:287] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.642s user 0.003s sys 0.007s
I20260806 20:16:02.435098 690 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:02.846415 690 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:02.788898 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 428809us
W20260806 20:16:02.464514 704 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.347s user 0.000s sys 0.001s
W20260806 20:16:02.635735 712 log.cc:924] Time spent T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260806 20:16:02.686771 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1731) took 1124 ms. Trace:
I20260806 20:16:02.730721 689 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:02.713703 715 log.cc:924] Time spent T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.292s user 0.000s sys 0.001s
I20260806 20:16:02.730161 32647 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:16:02.729993 388 consensus_queue.cc:1046] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20260806 20:16:02.508397 667 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:02.466401 313 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:02.534119 702 log.cc:924] Time spent T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.299s user 0.001s sys 0.000s
W20260806 20:16:02.619856 605 consensus_meta.cc:287] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.057s user 0.013s sys 0.000s
I20260806 20:16:02.896250 605 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:02.898505 605 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:02.898753 621 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:02.801301 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.802945 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:02.894963 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:02.900484 31415 raft_consensus.cc:1226] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
I20260806 20:16:02.901202 30710 raft_consensus.cc:2354] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7.
I20260806 20:16:02.902017 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.805825 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:02.903890 30863 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:02.807525 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:02.849565 568 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:02.849972 346 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:02.421926 547 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:02.809559 32733 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.810333 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.811098 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.507529 32672 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:02.434077 694 raft_consensus.cc:686] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:02.433812 618 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:02.817401 30870 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:16:02.948074 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.638159 599 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:02.825119 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.789351 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.730373 351 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:02.728973 691 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:16:02.647672 705 log.cc:924] Time spent T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.380s user 0.000s sys 0.001s
I20260806 20:16:02.954624 30863 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:02.827929 578 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:02.828099 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:02.814559 567 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:02.829208 717 log.cc:924] Time spent T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.302s user 0.001s sys 0.000s
I20260806 20:16:02.814574 662 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:02.966003 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:02.959930 578 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.822901 721 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:02.834736 30791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:02.838155 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1728) took 1306 ms. Trace:
I20260806 20:16:02.842325 608 raft_consensus.cc:3032] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:16:02.422698 631 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.381s
W20260806 20:16:02.843994 710 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.480s user 0.002s sys 0.000s
I20260806 20:16:02.821681 30871 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:02.855783 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.856106 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 185) took 1637 ms. Trace:
I20260806 20:16:02.858660 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.858179 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:02.714686 32443 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:02.824527 619 consensus_meta.cc:287] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.383s user 0.012s sys 0.002s
I20260806 20:16:02.910060 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:02.739775 597 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:02.790320 30783 consensus_meta.cc:287] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.522s user 0.009s sys 0.000s
I20260806 20:16:02.854506 683 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:02.872653 667 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:02.873894 664 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:16:02.932693 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 397810us
I20260806 20:16:02.976332 692 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:02.852649 30786 rpcz_store.cc:276] 0806 20:16:01.562347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:02.271614 (+709267us) service_pool.cc:227] Handling call
0806 20:16:02.348506 (+ 76892us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:02.364002 (+ 15496us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:16:02.407252 (+ 43250us) log.cc:842] Serialized 14 byte log entry
0806 20:16:02.570061 (+162809us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:02.591858 (+ 21797us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:02.607071 (+ 15213us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:02.640650 (+ 33579us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:02.674466 (+ 33816us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":269122,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":18472,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":22119,"thread_start_us":152485,"threads_started":2,"wal-append.queue_time_us":434643}
I20260806 20:16:03.059435 631 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:03.057305 31417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 441) took 1708 ms. Trace:
I20260806 20:16:03.069285 631 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.069794 664 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.986559 689 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.056560 518 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.070079 31417 rpcz_store.cc:276] 0806 20:16:01.349042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.352488 (+ 3446us) service_pool.cc:227] Handling call
0806 20:16:01.399311 (+ 46823us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:01.458318 (+ 59007us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:01.458353 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:01.458908 (+ 555us) log.cc:842] Serialized 26 byte log entry
0806 20:16:02.167226 (+708318us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:02.229266 (+ 62040us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:03.056829 (+827563us) spinlock_profiling.cc:243] Waited 56.8 ms on lock 0x7b54005d0554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:03.056883 (+ 54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:03.056938 (+ 55us) raft_consensus.cc:1671] finished
0806 20:16:03.056956 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:03.057240 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":130704640,"thread_start_us":696867,"threads_started":1,"wal-append.queue_time_us":658519}
I20260806 20:16:03.071909 664 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:02.984211 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:02.969170 690 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.058460 694 consensus_queue.cc:237] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.073163 31423 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:03.069545 30712 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:03.059067 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1369) took 1637 ms. Trace:
I20260806 20:16:03.077122 30709 rpcz_store.cc:276] 0806 20:16:01.421856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.421970 (+ 114us) service_pool.cc:227] Handling call
0806 20:16:01.424723 (+ 2753us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:02.385538 (+960815us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:03.058812 (+673274us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b5400030054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:03.059000 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2116170112}
I20260806 20:16:03.078469 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.078945 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.079738 30709 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:03.076362 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:03.079933 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:03.050294 703 log.cc:924] Time spent T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.682s user 0.003s sys 0.000s
I20260806 20:16:02.946575 30867 raft_consensus.cc:2379] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:03.093214 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.952036 30868 raft_consensus.cc:2379] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:02.952502 30676 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:03.076620 30789 rpcz_store.cc:276] 0806 20:16:01.218801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.423408 (+204607us) service_pool.cc:227] Handling call
0806 20:16:01.423714 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:01.423922 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:01.423945 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:01.424365 (+ 420us) log.cc:842] Serialized 26 byte log entry
0806 20:16:02.263521 (+839156us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:02.263560 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:02.263623 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:02.855891 (+592268us) raft_consensus.cc:1671] finished
0806 20:16:02.855911 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:02.856046 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":830214,"threads_started":1,"wal-append.queue_time_us":671824}
W20260806 20:16:03.095436 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 143016us
I20260806 20:16:03.072408 30783 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:03.096453 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1739) took 1079 ms. Trace:
I20260806 20:16:02.848865 640 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:02.856474 634 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:02.826979 701 log.cc:924] Time spent T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.755s user 0.001s sys 0.002s
I20260806 20:16:02.929769 425 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:03.051256 724 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
I20260806 20:16:02.956269 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:02.951172 564 consensus_meta.cc:287] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.525s user 0.007s sys 0.002s
I20260806 20:16:02.945843 32672 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:02.960709 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:02.921288 621 consensus_queue.cc:1046] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260806 20:16:02.963860 32733 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.050400 578 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:02.964171 351 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:02.965021 31418 raft_consensus.cc:2379] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 4: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:02.965758 662 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:02.829854 718 log.cc:924] Time spent T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.259s user 0.000s sys 0.001s
I20260806 20:16:02.971033 30791 raft_consensus.cc:595] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:16:02.970995 30790 rpcz_store.cc:276] 0806 20:16:01.531760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:02.166835 (+635075us) service_pool.cc:227] Handling call
0806 20:16:02.837994 (+671159us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b54002debd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:02.838096 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31330688}
I20260806 20:16:02.953537 691 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
W20260806 20:16:03.064491 613 consensus_meta.cc:287] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.697s user 0.007s sys 0.001s
I20260806 20:16:03.090276 30712 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:03.080495 30757 leader_election.cc:334] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.547s (SENT)
W20260806 20:16:02.924862 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 97544.2us
I20260806 20:16:03.100045 30867 raft_consensus.cc:2379] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:03.091710 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.167987 30757 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:03.168839 30871 raft_consensus.cc:2354] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7.
I20260806 20:16:03.166765 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.170365 30756 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:02.943722 30865 raft_consensus.cc:2454] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:16:03.092023 30869 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:02.973867 608 raft_consensus.cc:729] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:16:03.053573 730 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.116s user 0.001s sys 0.000s
W20260806 20:16:03.059350 506 consensus_meta.cc:287] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.709s user 0.007s sys 0.003s
I20260806 20:16:02.955955 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:03.012308 725 log.cc:924] Time spent T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260806 20:16:02.978219 548 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:03.110566 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.072645 667 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:03.073028 30861 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 1.
I20260806 20:16:03.127318 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 217) took 1772 ms. Trace:
I20260806 20:16:03.166831 691 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:03.111618 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.165792 32672 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:03.069274 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.111367 30870 raft_consensus.cc:2379] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:03.218246 30868 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:03.141129 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.139444 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:03.154978 30791 raft_consensus.cc:603] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:16:03.134636 690 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.158066 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.158831 31423 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:03.217908 30713 rpcz_store.cc:276] 0806 20:16:01.355373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.355947 (+ 574us) service_pool.cc:227] Handling call
0806 20:16:01.358691 (+ 2744us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:01.359070 (+ 379us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:01.359097 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:01.361396 (+ 2299us) log.cc:842] Serialized 26 byte log entry
0806 20:16:02.260928 (+899532us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:02.260964 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:02.261033 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:03.049643 (+788610us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:03.127041 (+ 77398us) raft_consensus.cc:1671] finished
0806 20:16:03.127059 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:03.127254 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":586,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":6787,"spinlock_wait_cycles":5307136,"thread_start_us":864445,"threads_started":1,"wal-append.queue_time_us":966404}
I20260806 20:16:03.129539 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33214 (request call id 595) took 3660 ms. Trace:
I20260806 20:16:03.220263 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.112118 30788 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:03.062564 618 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.058549 32574 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:03.219157 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 79822.9us
W20260806 20:16:03.187987 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 107520us
I20260806 20:16:03.154562 30835 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:03.188738 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.218554 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.189798 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.095723 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.228492 30867 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:03.227530 30676 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:16:03.227348 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 72881.5us
I20260806 20:16:03.096730 30783 rpcz_store.cc:276] 0806 20:16:02.016665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:02.031020 (+ 14355us) service_pool.cc:227] Handling call
0806 20:16:03.096390 (+1065370us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:03.194289 30711 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
W20260806 20:16:03.116276 711 log.cc:924] Time spent T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.594s user 0.003s sys 0.002s
I20260806 20:16:03.100721 685 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:03.209324 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:03.242327 30790 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:03.210188 31415 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:16:03.112212 31417 raft_consensus.cc:1226] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:03.211460 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.212050 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.243713 30869 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:03.244278 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.121630 640 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.146345 634 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:03.217645 518 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:03.220712 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.221036 30710 raft_consensus.cc:2379] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:03.220615 30784 rpcz_store.cc:276] 0806 20:15:59.493641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:00.039456 (+545815us) service_pool.cc:227] Handling call
0806 20:16:03.128674 (+3089218us) spinlock_profiling.cc:243] Waited 3.07 s on lock 0x7b54003b29d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4af0 00007f7151677f2a 00007f71565b4c6e 00007f714fcb7807 00007f714fcb7792 00007f714fcb76c1 00007f714fcb763c 00007f714fcb68f2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:03.129411 (+ 737us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7055373056}
I20260806 20:16:03.072283 631 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.224929 30712 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:03.230486 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:03.233997 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.234411 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.228665 30673 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:16:03.237020 30788 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53732 (request call id 188) took 1802 ms (client timeout 2134 ms). Trace:
W20260806 20:16:03.269825 30788 rpcz_store.cc:269] 0806 20:16:01.434771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.434994 (+ 223us) service_pool.cc:227] Handling call
0806 20:16:03.236849 (+1801855us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54003b29d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:03.236957 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3784032640}
I20260806 20:16:03.166226 564 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:02.980409 721 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.093580 683 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.245393 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.243608 30705 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
W20260806 20:16:03.242812 30757 leader_election.cc:334] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.857s (SENT)
I20260806 20:16:03.203236 663 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.255690 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 220) took 1715 ms. Trace:
I20260806 20:16:03.277998 30870 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
W20260806 20:16:03.247695 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 124053us
I20260806 20:16:03.244917 30711 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:03.114168 425 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.119036 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:03.261296 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.282440 30703 rpcz_store.cc:276] 0806 20:16:01.540402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.545027 (+ 4625us) service_pool.cc:227] Handling call
0806 20:16:01.545379 (+ 352us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:01.554118 (+ 8739us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:01.554146 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:01.562328 (+ 8182us) log.cc:842] Serialized 26 byte log entry
0806 20:16:02.130239 (+567911us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:02.130283 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:02.130363 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:02.843446 (+713083us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:03.255309 (+411863us) raft_consensus.cc:1671] finished
0806 20:16:03.255334 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:03.255613 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5162880,"thread_start_us":494615,"threads_started":1,"wal-append.queue_time_us":740320}
I20260806 20:16:03.215598 597 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:03.263423 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.256873 30835 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:03.266299 30754 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:03.268610 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.350178 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.350178 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.227921 548 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:03.222209 668 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:03.268767 30709 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:03.271621 30676 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:03.272457 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:03.375980 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 104434us
I20260806 20:16:03.376278 30710 raft_consensus.cc:2396] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 20:16:03.273047 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.376850 30869 raft_consensus.cc:2362] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7. Re-sending same reply.
I20260806 20:16:03.377848 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.378448 30869 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:03.274017 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.241243 32733 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:03.394944 32733 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:03.395437 32733 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:03.209039 385 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:03.274686 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.244836 32507 raft_consensus.cc:2781] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:16:03.247117 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.210227 506 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.227319 388 consensus_queue.cc:1046] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260806 20:16:03.287870 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:03.288069 30784 raft_consensus.cc:1261] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:16:03.445159 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:03.289112 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.267120 518 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:03.337970 30706 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:16:03.283970 30711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b"
dest_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:59126
W20260806 20:16:03.343086 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 56009.3us
I20260806 20:16:03.343266 30790 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
W20260806 20:16:03.343652 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 57025.8us
I20260806 20:16:03.291167 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.291976 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.350979 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.228971 30865 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:03.350219 30712 raft_consensus.cc:2396] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 20:16:03.277592 631 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:16:03.351323 31421 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 541) took 1812 ms. Trace:
W20260806 20:16:03.287477 682 consensus_meta.cc:287] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.856s user 0.006s sys 0.004s
I20260806 20:16:03.354907 30756 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:03.366336 662 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.356253 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.369026 30703 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:03.360580 621 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:03.282390 694 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:03.370350 30713 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
W20260806 20:16:03.370980 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 104765us
I20260806 20:16:03.354393 566 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:03.419065 30871 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:03.372865 548 raft_consensus.cc:686] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:03.366209 608 consensus_queue.cc:260] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:03.372612 668 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:03.396476 685 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:03.746423 30705 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:16:03.756240 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1564) took 1029 ms. Trace:
I20260806 20:16:03.756615 30705 rpcz_store.cc:276] 0806 20:16:02.726494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.278168 (+551674us) service_pool.cc:227] Handling call
0806 20:16:03.756156 (+477988us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:03.758672 31389 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:03.460952 30788 raft_consensus.cc:2379] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:03.747125 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.742542 30676 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:03.761569 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.468811 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.747759 31421 rpcz_store.cc:276] 0806 20:16:01.538599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.549341 (+ 10742us) service_pool.cc:227] Handling call
0806 20:16:01.549663 (+ 322us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:01.549892 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:01.549914 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:01.843782 (+293868us) spinlock_profiling.cc:243] Waited 7.21 ms on lock 0x7b2c00081bb8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:16:01.844267 (+ 485us) log.cc:842] Serialized 26 byte log entry
0806 20:16:02.344402 (+500135us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:02.344440 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:02.344507 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:03.115426 (+770919us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:03.351122 (+235696us) raft_consensus.cc:1671] finished
0806 20:16:03.351142 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:03.351269 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":101039232,"thread_start_us":499765,"threads_started":1,"wal-append.queue_time_us":541705}
I20260806 20:16:03.524464 30707 raft_consensus.cc:1226] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:16:03.749130 30711 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
W20260806 20:16:03.749722 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 396911us
I20260806 20:16:03.750265 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:03.557981 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 61603.2us
I20260806 20:16:03.751713 31415 raft_consensus.cc:1226] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
I20260806 20:16:03.763489 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.760443 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1570) took 1020 ms. Trace:
I20260806 20:16:03.547456 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:03.754921 30867 raft_consensus.cc:2379] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:03.764218 30791 raft_consensus.cc:2379] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:03.762740 30711 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:03.755039 30870 raft_consensus.cc:2379] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:03.765162 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.765707 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1556) took 1381 ms. Trace:
I20260806 20:16:03.766191 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.766424 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.767143 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.767601 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.768146 30756 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:03.748831 621 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:03.768944 621 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:03.768605 548 consensus_queue.cc:237] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.770645 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.768358 30867 raft_consensus.cc:2379] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:03.766086 30703 rpcz_store.cc:276] 0806 20:16:02.384635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.286407 (+901772us) service_pool.cc:227] Handling call
0806 20:16:03.357613 (+ 71206us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:03.765461 (+407848us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:03.765652 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:03.771955 30713 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260806 20:16:03.772854 30784 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:03.771835 30870 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:03.520264 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.764569 30712 rpcz_store.cc:276] 0806 20:16:02.739522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.268330 (+528808us) service_pool.cc:227] Handling call
0806 20:16:03.760375 (+492045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:03.774911 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:03.773203 30706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1561) took 1373 ms (client timeout 1827 ms). Trace:
W20260806 20:16:03.775875 30706 rpcz_store.cc:269] 0806 20:16:02.399354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:02.730034 (+330680us) service_pool.cc:227] Handling call
0806 20:16:03.773022 (+1042988us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b54002116d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:03.773134 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1004675840}
I20260806 20:16:03.758450 30871 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:03.775480 708 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:03.779347 30835 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:03.779620 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:03.710974 740 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260806 20:16:03.489380 737 log.cc:924] Time spent T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260806 20:16:03.542025 32642 consensus_meta.cc:287] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.799s user 0.010s sys 0.000s
W20260806 20:16:03.699043 567 consensus_meta.cc:287] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.730s user 0.008s sys 0.000s
I20260806 20:16:03.782709 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.426116 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.783437 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.780699 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:16:03.672577 738 log.cc:924] Time spent T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20260806 20:16:03.418663 32507 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:16:03.592094 741 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.208s user 0.000s sys 0.001s
I20260806 20:16:03.755909 685 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:03.674576 506 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.785164 597 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:03.785552 662 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:03.785836 719 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:03.785961 535 consensus_queue.cc:1046] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260806 20:16:03.788133 692 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:03.788735 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1380) took 1854 ms. Trace:
I20260806 20:16:03.766511 694 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:03.823596 719 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:03.825141 694 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:03.826437 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.783638 566 raft_consensus.cc:686] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:03.745404 425 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.791770 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.792411 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.794400 30783 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:03.424641 689 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:03.784709 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.516963 388 consensus_queue.cc:1225] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer 42cabca48db7403b8523a3382a046043 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 20:16:03.840608 30871 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:03.842407 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.844060 385 raft_consensus.cc:3032] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:03.844565 385 raft_consensus.cc:729] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:03.798754 30870 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:03.799633 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.846472 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.846143 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.847781 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.848577 30871 raft_consensus.cc:2354] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7.
I20260806 20:16:03.848934 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.849395 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.850425 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.851348 30788 raft_consensus.cc:2396] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 20:16:03.852159 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:03.853425 30788 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:03.637840 663 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:03.685070 313 consensus_meta.cc:287] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.774s user 0.006s sys 0.002s
I20260806 20:16:03.798794 30711 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.674952 564 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:03.760429 518 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:03.673319 683 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:03.770808 599 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:03.665729 721 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.806092 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:03.873354 30786 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
W20260806 20:16:03.874765 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 98541us
I20260806 20:16:03.878496 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1557) took 1493 ms. Trace:
I20260806 20:16:03.879068 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:03.879899 30788 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:03.807451 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:03.927186 30791 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:03.814663 32574 consensus_meta.cc:287] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.582s user 0.008s sys 0.000s
I20260806 20:16:03.818214 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:03.805745 567 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7.
I20260806 20:16:03.809094 32507 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:16:03.930114 32672 consensus_meta.cc:287] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.684s user 0.010s sys 0.000s
I20260806 20:16:03.930519 30863 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:03.930689 32672 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.760551 682 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:03.898288 566 consensus_queue.cc:237] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.933243 30713 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:03.934212 682 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:03.823789 30707 rpcz_store.cc:276] 0806 20:16:01.934605 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:02.006437 (+ 71832us) service_pool.cc:227] Handling call
0806 20:16:02.006758 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:03.762607 (+1755849us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:03.788503 (+ 25896us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400211bd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:03.788681 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3489951104}
I20260806 20:16:03.824676 692 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:03.800922 32642 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.783113 708 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.827569 597 raft_consensus.cc:686] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:16:03.827837 32443 consensus_meta.cc:287] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.721s user 0.010s sys 0.000s
I20260806 20:16:03.821346 662 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:03.789702 608 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:03.811307 685 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:03.832746 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1424) took 1114 ms. Trace:
I20260806 20:16:03.822816 506 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:03.930233 30786 raft_consensus.cc:1261] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:16:03.826880 719 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:03.866294 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:16:03.832191 691 consensus_meta.cc:287] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.576s user 0.009s sys 0.000s
I20260806 20:16:03.935359 30866 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:03.938743 30786 raft_consensus.cc:1261] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:16:03.940773 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:03.939559 30867 rpcz_store.cc:276] 0806 20:16:02.718234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.781938 (+1063704us) service_pool.cc:227] Handling call
0806 20:16:03.832646 (+ 50708us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:03.940267 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1285) took 1050 ms. Trace:
I20260806 20:16:03.936499 30790 raft_consensus.cc:2354] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3.
I20260806 20:16:03.946350 30863 rpcz_store.cc:276] 0806 20:16:02.890182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:02.955670 (+ 65488us) service_pool.cc:227] Handling call
0806 20:16:03.940097 (+984427us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54001e4454. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:03.940208 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1706058112}
I20260806 20:16:03.942836 30783 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:03.954200 30791 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:03.874014 30711 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.879135 30708 rpcz_store.cc:276] 0806 20:16:02.384890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.351953 (+967063us) service_pool.cc:227] Handling call
0806 20:16:03.355355 (+ 3402us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:03.355571 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:03.355595 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:03.356019 (+ 424us) log.cc:842] Serialized 26 byte log entry
0806 20:16:03.803002 (+446983us) raft_consensus.cc:1631] Marking committed up to 1
0806 20:16:03.803034 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:03.803078 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:03.878230 (+ 75152us) raft_consensus.cc:1671] finished
0806 20:16:03.878252 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:03.878427 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":446736,"threads_started":1,"wal-append.queue_time_us":475250}
I20260806 20:16:03.987460 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:59126 (request call id 1555) took 1603 ms. Trace:
I20260806 20:16:03.987871 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:03.987810 30711 rpcz_store.cc:276] 0806 20:16:02.384379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.283841 (+899462us) service_pool.cc:227] Handling call
0806 20:16:03.987397 (+703556us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:03.888108 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.808251 30784 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:03.837119 621 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:03.990870 30710 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:03.991251 30866 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:03.992100 30791 raft_consensus.cc:1226] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:16:03.875697 599 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:03.902230 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:03.938071 506 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:16:03.912564 32733 consensus_meta.cc:287] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.517s user 0.006s sys 0.002s
I20260806 20:16:03.845502 385 consensus_queue.cc:260] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.968794 351 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:03.870393 313 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.929874 30865 raft_consensus.cc:2379] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:03.932825 32672 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:03.855171 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1432) took 1073 ms. Trace:
W20260806 20:16:03.872458 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 84687.5us
I20260806 20:16:03.938550 30864 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:04.002007 30756 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:03.955552 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:16:03.934681 30757 leader_election.cc:334] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.660s (SENT)
I20260806 20:16:04.001245 30870 rpcz_store.cc:276] 0806 20:16:02.781891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.765905 (+984014us) service_pool.cc:227] Handling call
0806 20:16:03.855107 (+ 89202us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:04.003610 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 68970.6us
W20260806 20:16:04.004297 30757 leader_election.cc:334] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.688s (SENT)
I20260806 20:16:04.004374 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.002954 30867 raft_consensus.cc:2362] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7. Re-sending same reply.
I20260806 20:16:04.006181 30757 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:03.962631 30783 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:03.965538 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:03.972481 631 raft_consensus.cc:686] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:03.969861 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:04.021647 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:03.970700 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:16:03.971467 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 195537us
I20260806 20:16:03.978785 682 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:03.979362 32443 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:03.935734 692 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:03.938588 662 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:03.986474 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:04.030678 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 44285.5us
I20260806 20:16:03.900028 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:03.995785 506 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:03.993661 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 213595us
I20260806 20:16:04.035687 30787 raft_consensus.cc:1226] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:03.902321 750 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
W20260806 20:16:03.937234 667 consensus_meta.cc:287] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.704s user 0.010s sys 0.000s
I20260806 20:16:04.011391 30713 raft_consensus.cc:2396] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 20:16:04.012569 30835 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:04.022745 30863 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:03.973742 30786 raft_consensus.cc:1226] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:03.954947 691 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.025040 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:03.937424 597 consensus_queue.cc:237] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.050786 30867 raft_consensus.cc:2362] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3. Re-sending same reply.
W20260806 20:16:04.026728 30757 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.822s (SENT) [suppressed 25 similar messages]
I20260806 20:16:04.027360 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:04.022854 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 280431us
I20260806 20:16:03.940758 694 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:03.993887 599 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:03.902112 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.031548 31389 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:04.040020 30706 raft_consensus.cc:2379] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 3: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:04.043844 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.001605 634 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:04.046736 566 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:04.031487 719 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:04.035210 32647 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:04.055730 30865 raft_consensus.cc:2379] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:04.053323 30676 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:04.058815 31389 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:04.068198 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.069509 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.064860 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.081898 30867 raft_consensus.cc:2379] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:04.069099 30707 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:04.054551 30757 leader_election.cc:334] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.744s (SENT)
I20260806 20:16:04.040117 619 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:04.047636 32672 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:04.089740 30713 raft_consensus.cc:1226] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:04.084935 30757 leader_election.cc:334] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.778s (SENT)
I20260806 20:16:04.053104 385 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:04.001434 32733 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.003058 351 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:04.098519 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 45270.9us
W20260806 20:16:04.093430 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 51700.3us
I20260806 20:16:04.095099 566 raft_consensus.cc:686] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:04.096482 30832 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.096354 30834 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.102774 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:04.099206 30676 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:04.103626 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.104175 30676 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:04.056555 621 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:04.071604 716 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:04.103413 30834 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.105188 753 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:04.032963 682 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.108666 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:04.111881 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:04.043653 388 consensus_queue.cc:1046] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260806 20:16:04.032800 32443 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.052402 667 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.105919 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.088198 567 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:04.088016 535 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:04.116492 30712 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:04.092265 708 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:04.088519 619 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:04.126343 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:04.108945 30861 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:04.105690 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.101214 742 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:04.109781 716 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:04.112484 753 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:04.141716 30865 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:04.132552 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.072021 634 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.050474 750 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.130298 30867 raft_consensus.cc:2379] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:04.109285 32642 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:04.128141 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet aa7cfe759ca64ca89a0529c2d3da5bf7, TS d4181d4d907f477f9feddc44ba3b9207 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.106338 748 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:04.132007 30706 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 1 FOLLOWER]: Advancing to term 3
I20260806 20:16:04.150309 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:04.150555 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.151153 30786 raft_consensus.cc:2396] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:16:04.103174 719 consensus_queue.cc:1046] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260806 20:16:04.118357 690 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:04.112263 762 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:16:04.099649 32672 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.166383 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:04.168401 30870 raft_consensus.cc:2379] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:04.169317 32672 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.129268 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:04.167320 351 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:04.171309 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:04.149080 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.136090 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:04.173671 351 raft_consensus.cc:3032] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:04.146803 30792 raft_consensus.cc:1226] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
W20260806 20:16:04.146661 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet aa7cfe759ca64ca89a0529c2d3da5bf7, TS d4181d4d907f477f9feddc44ba3b9207 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.129055 621 consensus_queue.cc:1046] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260806 20:16:04.133519 619 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:04.151976 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.127560 749 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:04.139710 567 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.139313 708 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:04.131881 751 consensus_queue.cc:1046] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.132s
I20260806 20:16:04.117106 631 consensus_queue.cc:237] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.135242 535 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.340839 535 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:04.341557 30870 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:16:04.345909 313 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.346881 634 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:04.348352 567 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:04.272228 32733 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.169432 32507 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.349650 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:04.350073 30868 raft_consensus.cc:2379] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:04.350992 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1309) took 1138 ms. Trace:
I20260806 20:16:04.351433 668 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:04.352180 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:04.352947 31417 raft_consensus.cc:2354] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4.
W20260806 20:16:04.353758 30864 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1429) took 1627 ms (client timeout 1906 ms). Trace:
W20260806 20:16:04.354146 30864 rpcz_store.cc:269] 0806 20:16:02.726648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:02.734364 (+ 7716us) service_pool.cc:227] Handling call
0806 20:16:04.353605 (+1619241us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002fa8d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:04.353707 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2643121408}
W20260806 20:16:04.169976 757 log.cc:924] Time spent T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260806 20:16:04.143719 716 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:04.177351 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1447) took 1079 ms. Trace:
I20260806 20:16:04.274712 30791 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:04.249653 30706 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
W20260806 20:16:04.326154 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 144392us
I20260806 20:16:04.396490 30865 rpcz_store.cc:276] 0806 20:16:03.098191 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.151791 (+1053600us) service_pool.cc:227] Handling call
0806 20:16:04.177238 (+ 25447us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:04.325472 30757 leader_election.cc:334] T 616aae135c70498da00a9e8a1a61b1cd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.926s (SENT)
W20260806 20:16:04.397660 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 72282.2us
I20260806 20:16:04.329896 30786 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a439c5debb41440c913a50da91e6467f"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:04.398432 30786 raft_consensus.cc:595] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:16:04.398793 30786 raft_consensus.cc:603] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:16:04.256680 30861 raft_consensus.cc:1226] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:04.167541 690 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:04.286024 30788 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:04.422076 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1393) took 1074 ms. Trace:
I20260806 20:16:04.422443 30788 rpcz_store.cc:276] 0806 20:16:03.347601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.853991 (+506390us) service_pool.cc:227] Handling call
0806 20:16:04.422012 (+568021us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:04.424558 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 295355us
I20260806 20:16:04.426194 439 consensus_queue.cc:1046] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:16:04.427307 567 raft_consensus.cc:3032] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:16:04.429674 578 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:04.430883 578 raft_consensus.cc:686] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:04.431707 578 consensus_queue.cc:237] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.431906 662 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.258561 682 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:04.430263 30786 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:04.270782 750 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:04.338488 30785 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:04.351306 30868 rpcz_store.cc:276] 0806 20:16:03.212546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.233095 (+ 20549us) service_pool.cc:227] Handling call
0806 20:16:04.350830 (+1117735us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002c6d54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:04.350940 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2446228224}
I20260806 20:16:04.254833 691 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:04.356607 30867 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260806 20:16:04.358639 30866 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1442) took 1561 ms (client timeout 2077 ms). Trace:
I20260806 20:16:04.443540 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1425) took 1723 ms. Trace:
I20260806 20:16:04.270927 566 consensus_queue.cc:237] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.256641 32642 raft_consensus.cc:3032] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:16:04.393124 30832 leader_election.cc:398] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4.
I20260806 20:16:04.448943 30832 leader_election.cc:401] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 20:16:04.184674 351 raft_consensus.cc:729] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:16:04.449838 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 89064.9us
I20260806 20:16:04.151209 742 raft_consensus.cc:686] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
W20260806 20:16:04.365990 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 259466us
I20260806 20:16:04.271698 32443 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:04.270565 667 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:04.432621 388 raft_consensus.cc:3032] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:04.436658 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:04.257499 760 log.cc:924] Time spent T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
I20260806 20:16:04.443979 30867 rpcz_store.cc:276] 0806 20:16:02.720600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.336555 (+1615955us) service_pool.cc:227] Handling call
0806 20:16:04.356448 (+ 19893us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:04.443279 (+ 86831us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.443483 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:04.443737 30866 rpcz_store.cc:269] 0806 20:16:02.796941 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.234149 (+437208us) service_pool.cc:227] Handling call
0806 20:16:04.358485 (+1124336us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54002fadd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:04.358585 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1531115904}
I20260806 20:16:04.463706 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.464428 30866 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 2.
I20260806 20:16:04.465312 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.466344 30866 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:04.443470 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:04.467911 30783 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:04.469275 30835 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:04.470791 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:04.269346 685 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.152779 518 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.147002 753 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:04.434408 690 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:04.450129 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 346777us
I20260806 20:16:04.454917 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:04.450500 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 347888us
I20260806 20:16:04.461756 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:04.473661 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:04.476213 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:04.477697 30867 raft_consensus.cc:2379] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:04.476876 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:04.478245 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:04.480094 597 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:04.480548 597 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:04.446635 30676 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.481068 597 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:04.481817 30834 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.448195 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:04.427923 567 raft_consensus.cc:729] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:04.467563 30786 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:04.455564 694 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.455921 388 raft_consensus.cc:729] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:04.501621 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:04.460453 749 raft_consensus.cc:686] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:04.502449 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:04.448710 32642 raft_consensus.cc:729] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:04.503333 30756 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:04.432330 506 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.522024 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:04.522348 30835 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:04.478724 30703 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:04.451249 708 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4.
I20260806 20:16:04.506276 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:04.461350 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1423) took 1738 ms. Trace:
W20260806 20:16:04.501030 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 55642.4us
I20260806 20:16:04.510857 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:04.531349 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:04.513509 30866 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:04.531919 30867 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:04.510594 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:04.514859 30705 raft_consensus.cc:1226] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:04.517567 30790 raft_consensus.cc:1226] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:04.538221 30676 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.517140 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:04.452208 535 consensus_queue.cc:1046] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260806 20:16:04.523986 30834 leader_election.cc:398] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7.
I20260806 20:16:04.541553 30834 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:04.507018 613 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:16:04.528183 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:04.533205 31422 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:04.472512 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:04.543555 30834 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.575273 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:04.575843 30703 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:04.547915 30867 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:04.578238 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:04.546907 30754 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:04.581588 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:04.582093 30866 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:04.582800 755 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:04.583488 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:04.583690 685 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:04.584668 599 consensus_meta.cc:287] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.506s user 0.007s sys 0.001s
I20260806 20:16:04.585129 518 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.531087 30861 rpcz_store.cc:276] 0806 20:16:02.717624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.256103 (+1538479us) service_pool.cc:227] Handling call
0806 20:16:04.256475 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.443351 (+186876us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.455743 (+ 12392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:04.587316 30861 raft_consensus.cc:1261] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:16:04.587960 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1443) took 1790 ms. Trace:
I20260806 20:16:04.588296 30861 rpcz_store.cc:276] 0806 20:16:02.797299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.586567 (+1789268us) service_pool.cc:227] Handling call
0806 20:16:04.586903 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.587664 (+ 761us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.587860 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:04.515100 691 consensus_queue.cc:1046] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260806 20:16:04.575409 567 consensus_queue.cc:260] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.590051 749 consensus_queue.cc:237] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.591212 535 raft_consensus.cc:3032] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:04.591595 535 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:04.591606 567 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:04.593358 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:04.592363 535 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.590051 30712 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:04.590633 691 raft_consensus.cc:3032] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:04.535842 32574 raft_consensus.cc:2781] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:16:04.547626 30676 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:04.621378 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 73845.7us
W20260806 20:16:04.614517 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 71033.7us
I20260806 20:16:04.622592 30834 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:04.624480 773 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260806 20:16:04.624831 30834 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.616304 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:04.625787 30706 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:04.629792 30866 raft_consensus.cc:1261] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:16:04.630586 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1445) took 1779 ms. Trace:
I20260806 20:16:04.631330 30866 rpcz_store.cc:276] 0806 20:16:02.851150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.629048 (+1777898us) service_pool.cc:227] Handling call
0806 20:16:04.629393 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.630329 (+ 936us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.630530 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:04.636763 30866 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 20:16:04.636952 30706 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:04.632370 30834 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:04.687078 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 54799us
I20260806 20:16:04.688391 30834 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.689779 30834 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:04.691324 30834 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.692299 30834 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:04.614616 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 67815us
I20260806 20:16:04.637943 689 raft_consensus.cc:3032] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 20:16:04.638832 692 consensus_meta.cc:287] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.597s user 0.002s sys 0.007s
I20260806 20:16:04.641762 535 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:04.642768 506 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.643669 662 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.644240 32574 raft_consensus.cc:686] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:04.585091 599 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.629992 385 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:04.522996 767 consensus_queue.cc:1046] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.131s
I20260806 20:16:04.618357 691 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:04.577188 708 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:04.718585 439 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:04.625190 719 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:04.576949 682 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:04.619630 748 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:04.653537 597 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.611917 32642 consensus_queue.cc:260] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.540503 32672 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.116s
I20260806 20:16:04.515173 351 consensus_queue.cc:260] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.656133 690 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.597990 388 consensus_queue.cc:260] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.657061 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:04.805383 30791 raft_consensus.cc:2379] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:04.531055 742 consensus_queue.cc:237] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.659268 634 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:04.660521 30703 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:04.660894 613 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:04.661562 30871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec226f72b38a49529b88339beb36af46"
dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:60894
I20260806 20:16:04.661921 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.673580 621 raft_consensus.cc:3032] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:04.674041 564 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:04.677199 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1742) took 1827 ms. Trace:
I20260806 20:16:04.890555 30787 rpcz_store.cc:276] 0806 20:16:02.850062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:02.908140 (+ 58078us) service_pool.cc:227] Handling call
0806 20:16:02.937490 (+ 29350us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.036106 (+1098616us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.676942 (+640836us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400012cd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:04.677112 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2525222784}
I20260806 20:16:04.676016 716 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.627408 32507 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:04.681048 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:04.694182 30754 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:04.694677 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 254433us
I20260806 20:16:04.696161 30756 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:04.718366 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1287) took 1801 ms. Trace:
I20260806 20:16:04.688534 683 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:04.717828 689 raft_consensus.cc:729] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:04.625625 32443 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:04.718776 777 raft_consensus.cc:2741] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 20:16:04.797797 30792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1747) took 1649 ms. Trace:
I20260806 20:16:04.799156 30788 raft_consensus.cc:1226] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
W20260806 20:16:04.792137 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 99887.9us
W20260806 20:16:04.845598 668 consensus_queue.cc:1159] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "42cabca48db7403b8523a3382a046043" responder_term: 7 status { last_received { term: 6 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 20:16:04.849946 388 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:04.949857 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 253775us
I20260806 20:16:04.843619 30867 raft_consensus.cc:2354] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6.
I20260806 20:16:04.924634 31423 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:16:04.720930 692 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.720434 753 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:04.849009 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1291) took 1930 ms (client timeout 2130 ms). Trace:
W20260806 20:16:04.957439 30869 rpcz_store.cc:269] 0806 20:16:02.918345 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.752148 (+833803us) service_pool.cc:227] Handling call
0806 20:16:04.848777 (+1096629us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b54002fbcd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:04.848941 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":595183872}
I20260806 20:16:04.791570 548 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:04.960528 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:16:04.893363 619 consensus_meta.cc:287] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.538s user 0.004s sys 0.004s
I20260806 20:16:04.794924 682 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:04.983246 682 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:04.983526 690 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.886656 30712 raft_consensus.cc:1226] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:04.887583 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 225756us
W20260806 20:16:04.898228 778 log.cc:924] Time spent T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260806 20:16:04.986017 30707 raft_consensus.cc:1226] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:04.986096 30676 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.889176 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:04.986883 31419 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:04.901151 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1282) took 2348 ms. Trace:
I20260806 20:16:04.851866 30861 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260806 20:16:04.988348 30676 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:04.917750 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 223637us
W20260806 20:16:04.961378 798 log.cc:924] Time spent T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.053s user 0.001s sys 0.003s
I20260806 20:16:04.990414 30754 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:04.990774 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.987481 30863 rpcz_store.cc:276] 0806 20:16:02.553118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.230025 (+1676907us) service_pool.cc:227] Handling call
0806 20:16:04.329231 (+ 99206us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:04.336436 (+ 7205us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:04.336466 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:04.355207 (+ 18741us) log.cc:842] Serialized 26 byte log entry
0806 20:16:04.559566 (+204359us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:04.808765 (+249199us) log.cc:842] Serialized 14 byte log entry
0806 20:16:04.844647 (+ 35882us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.844722 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:04.900925 (+ 56203us) raft_consensus.cc:1671] finished
0806 20:16:04.900949 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:04.901086 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":346161,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":43,"raft.queue_time_us":47007,"raft.run_cpu_time_us":8551,"raft.run_wall_time_us":35596,"thread_start_us":385508,"threads_started":2,"wal-append.queue_time_us":430359}
I20260806 20:16:04.991950 30676 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.993074 30676 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.993202 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1278) took 2567 ms. Trace:
I20260806 20:16:04.993613 30865 raft_consensus.cc:1261] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260806 20:16:04.790438 30757 leader_election.cc:334] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.130s (SENT)
W20260806 20:16:04.982970 30785 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1380) took 1638 ms (client timeout 2113 ms). Trace:
I20260806 20:16:04.995133 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:04.994765 30757 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:04.790799 385 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:04.961961 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:04.966992 506 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260806 20:16:04.961619 797 log.cc:924] Time spent T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.056s user 0.001s sys 0.001s
I20260806 20:16:04.892280 32574 consensus_queue.cc:237] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:04.956403 32443 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:04.978231 597 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:04.904744 631 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:04.905869 599 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.985560 753 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.808593 566 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:04.996855 776 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:04.916814 30866 rpcz_store.cc:276] 0806 20:16:02.917189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.636121 (+1718932us) service_pool.cc:227] Handling call
0806 20:16:04.636511 (+ 390us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:04.717994 (+ 81483us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.718291 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.025202 30866 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260806 20:16:05.026000 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1296) took 2089 ms. Trace:
I20260806 20:16:05.026356 30866 rpcz_store.cc:276] 0806 20:16:02.937030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.024500 (+2087470us) service_pool.cc:227] Handling call
0806 20:16:05.024928 (+ 428us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:05.025699 (+ 771us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.025938 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:04.993630 30870 rpcz_store.cc:276] 0806 20:16:02.426114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.168975 (+1742861us) service_pool.cc:227] Handling call
0806 20:16:04.169341 (+ 366us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:04.341970 (+172629us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.992944 (+650974us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54002413d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:04.993139 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":395393280}
I20260806 20:16:04.954875 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:04.953774 30792 rpcz_store.cc:276] 0806 20:16:03.148111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.156301 (+ 8190us) service_pool.cc:227] Handling call
0806 20:16:03.156650 (+ 349us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.326642 (+1169992us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.797500 (+470858us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b54003a0cd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:04.797713 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2272380288}
I20260806 20:16:05.004848 751 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:05.010584 32443 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:04.983011 619 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.023293 689 consensus_queue.cc:260] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:05.041383 689 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:05.042346 716 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:05.048765 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1298) took 2102 ms. Trace:
I20260806 20:16:05.049247 30870 rpcz_store.cc:276] 0806 20:16:02.946220 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.028056 (+2081836us) service_pool.cc:227] Handling call
0806 20:16:05.048196 (+ 20140us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.048431 (+ 235us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:05.048458 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:05.048501 (+ 43us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:16:05.048542 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.048590 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.048698 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.050743 30870 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:05.051385 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1300) took 2100 ms. Trace:
I20260806 20:16:05.051687 30870 rpcz_store.cc:276] 0806 20:16:02.950953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.050192 (+2099239us) service_pool.cc:227] Handling call
0806 20:16:05.050557 (+ 365us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.051138 (+ 581us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.051330 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.052439 32642 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:05.053500 30783 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:05.054631 619 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:04.887116 621 raft_consensus.cc:729] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:16:04.956496 795 log.cc:924] Time spent T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.053s user 0.004s sys 0.000s
I20260806 20:16:05.004663 30784 raft_consensus.cc:1226] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:05.058743 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1385) took 1713 ms. Trace:
I20260806 20:16:04.807873 748 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:04.956955 784 log.cc:924] Time spent T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.067s user 0.002s sys 0.002s
I20260806 20:16:05.006320 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1295) took 2076 ms. Trace:
I20260806 20:16:05.004106 690 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:04.957026 767 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:04.953934 788 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:04.887518 691 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.001811 749 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:05.004659 754 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:04.997254 613 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:05.004431 32733 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:05.008661 31417 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:05.009675 548 raft_consensus.cc:2741] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:16:04.994460 658 consensus_queue.cc:1046] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260806 20:16:05.065507 716 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:05.074738 788 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:05.089993 608 raft_consensus.cc:3032] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:05.090590 608 raft_consensus.cc:729] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:05.091444 608 consensus_queue.cc:260] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.092773 608 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:05.093314 385 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:05.079206 767 raft_consensus.cc:686] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:05.094086 788 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:05.096807 716 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:05.107932 781 log.cc:924] Time spent T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.487s user 0.000s sys 0.002s
I20260806 20:16:05.002727 30676 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:05.143288 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 140484us
I20260806 20:16:05.146348 30676 leader_election.cc:398] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3.
I20260806 20:16:05.146783 30676 leader_election.cc:401] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:16:05.001204 30703 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:05.155201 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1408) took 1107 ms. Trace:
I20260806 20:16:05.155772 30707 rpcz_store.cc:276] 0806 20:16:04.047201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.082802 (+ 35601us) service_pool.cc:227] Handling call
0806 20:16:04.101246 (+ 18444us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.986518 (+885272us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.154567 (+168049us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b54000901d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:05.154967 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2034419968}
I20260806 20:16:05.158214 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:05.159052 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1393) took 1944 ms. Trace:
I20260806 20:16:05.157377 30602 catalog_manager.cc:5495] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:05.159492 30705 rpcz_store.cc:276] 0806 20:16:03.214903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.757471 (+542568us) service_pool.cc:227] Handling call
0806 20:16:03.757832 (+ 361us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.533645 (+775813us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.158354 (+624709us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b54001b5154. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:05.158811 (+ 457us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1740548352}
I20260806 20:16:04.997997 30706 raft_consensus.cc:1226] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:16:05.161757 30602 catalog_manager.cc:5495] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 3 to 4, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 4 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:16:05.163235 30706 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:05.059177 30789 rpcz_store.cc:276] 0806 20:16:03.345418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.883696 (+538278us) service_pool.cc:227] Handling call
0806 20:16:03.884024 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.451835 (+567811us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:04.451866 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:04.451906 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:04.451936 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.058490 (+606554us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b540026f554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:05.058536 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.058666 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3109,"raft.queue_time_us":3522,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"spinlock_wait_cycles":1297836416}
I20260806 20:16:05.009719 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:05.166853 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1396) took 1950 ms. Trace:
I20260806 20:16:05.167407 30709 rpcz_store.cc:276] 0806 20:16:03.216261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.769736 (+553475us) service_pool.cc:227] Handling call
0806 20:16:03.770045 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:03.874045 (+104000us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:16:04.017298 (+143253us) log.cc:842] Serialized 14 byte log entry
0806 20:16:04.360583 (+343285us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b18001a3140. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f714e58e990 00007f714e750864 00007f714e750155 00007f71516673d0 00007f7151679576 00007f715168a0d7 00007f715168a058
0806 20:16:04.984819 (+624236us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b18001a3140. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f714e58e990 00007f714e750864 00007f714e750155 00007f7155044167 00007f7151679807 00007f7151688f0f 00007f7151688e88
0806 20:16:04.985150 (+ 331us) log.cc:842] Serialized 14 byte log entry
0806 20:16:04.985330 (+ 180us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:05.000858 (+ 15528us) log.cc:842] Serialized 26 byte log entry
0806 20:16:05.001017 (+ 159us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:05.001051 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.001130 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:05.166384 (+165254us) raft_consensus.cc:1671] finished
0806 20:16:05.166428 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.166667 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":138012,"prepare.queue_time_us":882048,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":73,"raft.queue_time_us":311382,"raft.run_cpu_time_us":447,"raft.run_wall_time_us":45479,"spinlock_wait_cycles":42572800,"thread_start_us":197713,"threads_started":3,"wal-append.queue_time_us":98293}
I20260806 20:16:05.001919 30710 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:05.169904 30789 raft_consensus.cc:1261] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:16:05.061486 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1749) took 1706 ms. Trace:
I20260806 20:16:04.851357 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:05.059846 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1757) took 1070 ms. Trace:
I20260806 20:16:05.166780 30783 raft_consensus.cc:1261] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 20:16:05.172871 30602 catalog_manager.cc:5495] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 4 to 7, leader changed from 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 7 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:05.030642 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1439) took 2234 ms. Trace:
I20260806 20:16:05.295166 30865 rpcz_store.cc:276] 0806 20:16:02.796058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.397352 (+1601294us) service_pool.cc:227] Handling call
0806 20:16:04.993265 (+595913us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.993986 (+ 721us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.030429 (+ 36443us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b54002413d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:05.030577 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1257819520}
I20260806 20:16:05.062918 30863 rpcz_store.cc:276] 0806 20:16:02.923207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.998717 (+2075510us) service_pool.cc:227] Handling call
0806 20:16:04.999027 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.999229 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:04.999253 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:04.999292 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:04.999330 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.999380 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:04.999476 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.299563 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1303) took 2340 ms. Trace:
I20260806 20:16:05.299991 30863 rpcz_store.cc:276] 0806 20:16:02.959339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.298122 (+2338783us) service_pool.cc:227] Handling call
0806 20:16:05.298622 (+ 500us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.299057 (+ 435us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:05.299079 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:05.299140 (+ 61us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:05.299187 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.299240 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.299367 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.000013 30754 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:05.302564 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 302455us
W20260806 20:16:04.998658 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 208259us
I20260806 20:16:05.304817 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:04.995195 30785 rpcz_store.cc:269] 0806 20:16:03.344074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.782446 (+438372us) service_pool.cc:227] Handling call
0806 20:16:04.982799 (+1200353us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b540026e154. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:04.982906 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1235279744}
I20260806 20:16:05.261838 30790 rpcz_store.cc:276] 0806 20:16:03.355370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.964039 (+608669us) service_pool.cc:227] Handling call
0806 20:16:03.978445 (+ 14406us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.533939 (+555494us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.061203 (+527264us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54000573d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:05.061396 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1239355776}
I20260806 20:16:05.294131 385 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:05.311039 30785 raft_consensus.cc:1226] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:05.310849 385 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.313923 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:05.028492 30792 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:05.316337 30792 raft_consensus.cc:595] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:05.188354 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1444) took 2358 ms. Trace:
I20260806 20:16:05.191038 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1283) took 2405 ms. Trace:
W20260806 20:16:05.309494 30757 leader_election.cc:334] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.137s (SENT)
I20260806 20:16:05.192082 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1441) took 2395 ms. Trace:
I20260806 20:16:05.192654 30871 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:16:05.318799 30757 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:05.319186 30871 raft_consensus.cc:3037] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:05.321035 30757 leader_election.cc:334] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.143s (SENT)
W20260806 20:16:05.195374 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 240450us
I20260806 20:16:05.318410 30864 rpcz_store.cc:276] 0806 20:16:02.785859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.354937 (+1569078us) service_pool.cc:227] Handling call
0806 20:16:04.355213 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.355401 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:16:04.683813 (+328412us) log.cc:842] Serialized 14 byte log entry
0806 20:16:04.885478 (+201665us) log.cc:842] Serialized 14 byte log entry
0806 20:16:05.189765 (+304287us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b180032e7e0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f714e58e990 00007f714e750864 00007f714e750155 00007f71516673d0 00007f7151679576 00007f715168a0d7 00007f715168a058
0806 20:16:05.190223 (+ 458us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:05.190344 (+ 121us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:05.190481 (+ 137us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.190616 (+ 135us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.190826 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":366020,"prepare.queue_time_us":607109,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":93,"raft.queue_time_us":373240,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":5944,"spinlock_wait_cycles":2545920,"thread_start_us":485052,"threads_started":2,"wal-append.queue_time_us":219931}
I20260806 20:16:05.323877 30834 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:05.210224 31419 raft_consensus.cc:1226] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:05.212805 751 consensus_queue.cc:1046] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:16:05.222440 32507 consensus_queue.cc:1046] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260806 20:16:05.101943 692 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:05.325981 385 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:05.327631 30865 raft_consensus.cc:1261] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:05.328521 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1302) took 2374 ms. Trace:
I20260806 20:16:05.326634 385 raft_consensus.cc:3032] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:05.330250 385 raft_consensus.cc:729] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:05.329237 30868 rpcz_store.cc:276] 0806 20:16:02.796668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.438227 (+1641559us) service_pool.cc:227] Handling call
0806 20:16:04.451077 (+ 12850us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:04.453625 (+ 2548us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:04.453659 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:04.462467 (+ 8808us) log.cc:842] Serialized 26 byte log entry
0806 20:16:04.795513 (+333046us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:04.795563 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:04.795627 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:05.191587 (+395960us) raft_consensus.cc:1671] finished
0806 20:16:05.191628 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.191870 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3916,"raft.queue_time_us":2259,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":5902,"thread_start_us":295531,"threads_started":1,"wal-append.queue_time_us":425727}
I20260806 20:16:05.331537 385 consensus_queue.cc:260] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.325378 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1305) took 2350 ms. Trace:
I20260806 20:16:05.325806 751 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:16:05.106988 748 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:05.109239 787 raft_consensus.cc:3032] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 20:16:05.356462 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 130814us
W20260806 20:16:05.248281 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 97189.5us
I20260806 20:16:05.357697 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.256134 30787 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:05.361047 776 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:05.361336 716 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:05.362032 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:05.213348 775 consensus_queue.cc:1046] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260806 20:16:05.258062 753 consensus_queue.cc:1046] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:16:05.257107 690 consensus_queue.cc:1035] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 20:16:05.266579 767 consensus_queue.cc:237] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.270360 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:05.270639 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:05.272068 668 consensus_queue.cc:1046] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:16:05.273278 578 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:05.278560 750 consensus_queue.cc:1046] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:16:05.186726 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1752) took 1728 ms. Trace:
I20260806 20:16:05.328917 30865 rpcz_store.cc:276] 0806 20:16:02.954408 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.296697 (+2342289us) service_pool.cc:227] Handling call
0806 20:16:05.297409 (+ 712us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.328173 (+ 30764us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.328437 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.215767 755 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:05.333185 385 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:05.339972 30864 rpcz_store.cc:276] 0806 20:16:02.975461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.323419 (+2347958us) service_pool.cc:227] Handling call
0806 20:16:05.323952 (+ 533us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.325016 (+ 1064us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:05.325045 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:05.325107 (+ 62us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:05.325144 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.325200 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.325321 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":126,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":150}
I20260806 20:16:05.355738 30863 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:05.362218 506 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:05.103696 621 consensus_queue.cc:260] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.257086 749 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 2. Current term is 3.
I20260806 20:16:05.362514 799 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:05.329028 631 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260806 20:16:05.356910 787 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:05.294168 30788 rpcz_store.cc:276] 0806 20:16:03.989046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.424043 (+434997us) service_pool.cc:227] Handling call
0806 20:16:04.424319 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:04.954156 (+529837us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.059567 (+105411us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b540026e654. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:05.059772 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":861580672}
W20260806 20:16:05.357309 31389 leader_election.cc:334] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.009s (SENT)
I20260806 20:16:05.356956 30757 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:05.317622 30861 rpcz_store.cc:276] 0806 20:16:02.829351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.589177 (+1759826us) service_pool.cc:227] Handling call
0806 20:16:04.589645 (+ 468us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:04.987721 (+398076us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.187647 (+199926us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54000b1154. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:05.188112 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":470950784}
I20260806 20:16:05.368543 748 raft_consensus.cc:3032] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:05.378367 30784 rpcz_store.cc:276] 0806 20:16:03.458599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:03.990928 (+532329us) service_pool.cc:227] Handling call
0806 20:16:03.991202 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.058649 (+1067447us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.186168 (+127519us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54003d4ad4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:05.186528 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2326670080}
I20260806 20:16:05.388509 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1304) took 2416 ms. Trace:
W20260806 20:16:05.462425 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 141478us
W20260806 20:16:05.434888 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 49086.6us
I20260806 20:16:05.466714 30865 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:05.416548 30710 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:05.513675 30863 rpcz_store.cc:276] 0806 20:16:02.969883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.300923 (+2331040us) service_pool.cc:227] Handling call
0806 20:16:05.301404 (+ 481us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.384741 (+ 83337us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.386367 (+ 1626us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:05.457597 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 133846us
I20260806 20:16:05.485742 30864 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:16:05.401757 31389 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:05.449828 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 541832us
I20260806 20:16:05.528728 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1446) took 2430 ms. Trace:
I20260806 20:16:05.529141 30865 rpcz_store.cc:276] 0806 20:16:03.097899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.384347 (+2286448us) service_pool.cc:227] Handling call
0806 20:16:05.466400 (+ 82053us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.518191 (+ 51791us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.528647 (+ 10456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.452710 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:05.523590 776 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:05.535017 776 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:05.535224 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 178035us
I20260806 20:16:05.535533 30871 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.538038 771 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:05.538609 30865 raft_consensus.cc:1226] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:16:05.539268 567 raft_consensus.cc:2741] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 20:16:05.540005 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1313) took 2119 ms. Trace:
I20260806 20:16:05.537102 31389 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.542162 716 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:05.542716 716 raft_consensus.cc:686] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:05.540875 30865 rpcz_store.cc:276] 0806 20:16:03.420115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.537752 (+2117637us) service_pool.cc:227] Handling call
0806 20:16:05.538240 (+ 488us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.539420 (+ 1180us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.539861 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.450656 30703 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:05.490674 30756 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:05.528249 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1449) took 2419 ms. Trace:
I20260806 20:16:05.546970 30864 rpcz_store.cc:276] 0806 20:16:03.098673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.387442 (+2288769us) service_pool.cc:227] Handling call
0806 20:16:05.485254 (+ 97812us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.516745 (+ 31491us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.518380 (+ 1635us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.486974 30861 raft_consensus.cc:1226] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 4. Current term is 5. Ops: []
W20260806 20:16:05.443778 535 consensus_meta.cc:287] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.658s user 0.009s sys 0.005s
I20260806 20:16:05.458437 719 raft_consensus.cc:3032] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 20:16:05.465052 388 consensus_meta.cc:287] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.506s user 0.008s sys 0.003s
I20260806 20:16:05.560142 30786 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:16:05.402825 506 raft_consensus.cc:686] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:05.543726 716 consensus_queue.cc:237] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:05.561982 506 consensus_queue.cc:237] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.554196 548 raft_consensus.cc:2741] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:16:05.554607 719 raft_consensus.cc:729] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:05.488361 351 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:05.466619 599 raft_consensus.cc:3032] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:05.455401 691 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:05.513336 748 raft_consensus.cc:729] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:05.463227 667 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:16:05.573614 30757 leader_election.cc:334] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.151s (SENT)
I20260806 20:16:05.578315 30757 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:05.462883 32672 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.110s
I20260806 20:16:05.581224 32672 raft_consensus.cc:3032] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:05.581631 32672 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:05.582669 32672 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:05.584188 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:05.465739 578 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.584960 30789 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:16:05.579523 30711 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:16:05.452531 788 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:05.577824 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1453) took 2346 ms. Trace:
I20260806 20:16:05.449101 32443 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.464859 682 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.395076 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:05.590858 30676 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.457010 733 consensus_queue.cc:1046] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.119s
I20260806 20:16:05.593573 30792 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:05.577355 388 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:05.573033 599 raft_consensus.cc:729] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:05.611974 535 consensus_queue.cc:1046] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:16:05.595309 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:05.613641 535 raft_consensus.cc:3032] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:16:05.613385 599 consensus_queue.cc:260] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.614073 30789 raft_consensus.cc:1261] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:16:05.614539 31417 raft_consensus.cc:2379] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 8: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:05.614915 599 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:05.613693 608 consensus_meta.cc:287] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.520s user 0.013s sys 0.000s
I20260806 20:16:05.619555 668 raft_consensus.cc:3032] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:05.620043 668 raft_consensus.cc:729] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:05.620898 788 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:05.478825 787 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:05.600950 578 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:05.601747 30785 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 20:16:05.601572 30757 leader_election.cc:334] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.195s (SENT)
I20260806 20:16:05.602834 30863 raft_consensus.cc:1226] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
I20260806 20:16:05.623724 30757 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:05.624516 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:05.625351 30790 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:05.602506 748 consensus_queue.cc:260] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.587021 30834 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:05.591854 30861 rpcz_store.cc:276] 0806 20:16:03.231574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.486177 (+2254603us) service_pool.cc:227] Handling call
0806 20:16:05.486617 (+ 440us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.553628 (+ 67011us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.576412 (+ 22784us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.584204 32672 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:05.653637 30787 raft_consensus.cc:595] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:05.624038 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1455) took 1174 ms. Trace:
I20260806 20:16:05.643659 30791 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
W20260806 20:16:05.640084 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 149659us
I20260806 20:16:05.640054 30871 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:05.684036 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1454) took 1234 ms. Trace:
I20260806 20:16:05.627383 30868 raft_consensus.cc:1226] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:05.643117 31388 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:05.687371 30865 rpcz_store.cc:276] 0806 20:16:04.449473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.568491 (+1119018us) service_pool.cc:227] Handling call
0806 20:16:05.609737 (+ 41246us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.683584 (+ 73847us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:05.683614 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:05.683687 (+ 73us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:05.683742 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.683820 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.683968 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.690541 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:05.643482 631 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:05.673134 30864 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:05.683784 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:05.614114 535 raft_consensus.cc:729] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:05.690050 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:05.642896 682 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:05.696434 776 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:05.697307 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1297) took 2758 ms. Trace:
I20260806 20:16:05.657922 32642 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:05.671674 30787 raft_consensus.cc:603] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 20:16:05.623983 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 47560.6us
I20260806 20:16:05.648746 608 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:05.702824 30783 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:05.672432 30863 rpcz_store.cc:276] 0806 20:16:04.449788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.572442 (+1122654us) service_pool.cc:227] Handling call
0806 20:16:05.602534 (+ 30092us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.623806 (+ 21272us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.623984 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:05.705669 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:60894 (request call id 1292) took 2772 ms. Trace:
I20260806 20:16:05.699611 631 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:16:05.709972 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 101422us
I20260806 20:16:05.653770 32507 consensus_queue.cc:1046] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260806 20:16:05.715737 30871 rpcz_store.cc:276] 0806 20:16:02.918591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.661383 (+1742792us) service_pool.cc:227] Handling call
0806 20:16:05.625999 (+964616us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b5400240ed4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f7151666da7 00007f71565b2ee0 00007f714fcb0784 00007f714fcb0712 00007f714fcb0641 00007f714fcb05bc 00007f714fcaf872 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:05.690591 (+ 64592us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":707282304}
I20260806 20:16:05.647912 689 consensus_queue.cc:1035] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260806 20:16:05.646828 32443 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:05.644069 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:05.722273 30785 raft_consensus.cc:1261] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:05.722765 32647 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:05.759150 32647 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:05.759562 32647 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:05.760524 682 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:05.724859 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1399) took 1269 ms. Trace:
I20260806 20:16:05.761626 30791 rpcz_store.cc:276] 0806 20:16:04.455850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.806106 (+350256us) service_pool.cc:227] Handling call
0806 20:16:05.724656 (+918550us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54001a1b54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:05.724794 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":600740608}
I20260806 20:16:05.725649 32507 raft_consensus.cc:3032] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:05.762732 32507 raft_consensus.cc:729] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:05.763772 32507 consensus_queue.cc:260] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.761035 30792 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:05.765630 535 consensus_queue.cc:260] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:05.766747 30835 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.710640 30866 rpcz_store.cc:276] 0806 20:16:02.938409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.027105 (+2088696us) service_pool.cc:227] Handling call
0806 20:16:05.027478 (+ 373us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:05.266290 (+238812us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:05.266337 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:05.267085 (+ 748us) log.cc:842] Serialized 26 byte log entry
0806 20:16:05.376096 (+109011us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:05.460208 (+ 84112us) log.cc:842] Serialized 14 byte log entry
0806 20:16:05.460397 (+ 189us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.460496 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:05.696954 (+236458us) raft_consensus.cc:1671] finished
0806 20:16:05.696976 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.697122 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":167583,"prepare.queue_time_us":84448,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":199958,"raft.run_cpu_time_us":434,"raft.run_wall_time_us":436,"thread_start_us":163545,"threads_started":2,"wal-append.queue_time_us":355612}
I20260806 20:16:05.767493 535 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:05.767663 682 raft_consensus.cc:2781] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:05.646366 719 consensus_queue.cc:260] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.768435 682 raft_consensus.cc:686] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:05.768738 719 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:05.732748 613 raft_consensus.cc:3032] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:05.731735 30602 catalog_manager.cc:5495] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 2 to 6, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 6 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
W20260806 20:16:05.769706 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 221003us
I20260806 20:16:05.770633 30834 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:05.770741 30602 catalog_manager.cc:5495] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 4 to 5, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 5 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:05.722515 30790 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
W20260806 20:16:05.731600 30601 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34006 (request call id 126) took 11712 ms (client timeout 15000 ms). Trace:
I20260806 20:16:05.734767 658 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:16:05.735415 755 consensus_queue.cc:1035] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:05.736317 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1299) took 2749 ms. Trace:
I20260806 20:16:05.690408 799 consensus_queue.cc:1046] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260806 20:16:05.738276 30756 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.739465 787 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:05.736217 809 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:16:05.621201 668 consensus_queue.cc:260] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.719641 631 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:05.736222 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:05.749631 30676 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:05.751704 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1301) took 2799 ms. Trace:
I20260806 20:16:05.727319 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1460) took 1273 ms. Trace:
I20260806 20:16:05.801690 30864 rpcz_store.cc:276] 0806 20:16:04.454066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.568237 (+1114171us) service_pool.cc:227] Handling call
0806 20:16:05.608838 (+ 40601us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.699878 (+ 91040us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.727055 (+ 27177us) spinlock_profiling.cc:243] Waited 49.5 ms on lock 0x7b54002a0b54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:05.727251 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113759616}
I20260806 20:16:05.769673 613 raft_consensus.cc:729] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:05.774574 30835 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.775691 808 raft_consensus.cc:2726] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:05.651716 812 consensus_queue.cc:1046] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
W20260806 20:16:05.779000 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b060012888ff4d76b254b43d1f1959bb, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:05.815299 815 log.cc:924] Time spent T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.081s user 0.002s sys 0.000s
I20260806 20:16:05.754019 776 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:05.772560 30601 rpcz_store.cc:269] 0806 20:15:54.019016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:15:54.021607 (+ 2591us) service_pool.cc:227] Handling call
0806 20:16:05.731342 (+11709735us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:16:02.072287 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:16:02.158036 (+ 85749us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:16:02.160362 (+ 2326us) write_op.cc:435] Acquired schema lock
0806 20:16:02.163779 (+ 3417us) tablet.cc:660] Decoding operations
0806 20:16:02.378403 (+214624us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:16:02.380997 (+ 2594us) write_op.cc:641] Partition lock acquired for write op
0806 20:16:02.386397 (+ 5400us) tablet.cc:683] Acquiring locks for 6 operations
0806 20:16:02.664263 (+277866us) tablet.cc:699] Row locks acquired
0806 20:16:02.675969 (+ 11706us) write_op.cc:260] PREPARE: finished
0806 20:16:02.719192 (+ 43223us) write_op.cc:270] Start()
0806 20:16:02.782654 (+ 63462us) write_op.cc:276] Timestamp: P: 1786047362715425 usec, L: 0
0806 20:16:02.782685 (+ 31us) op_driver.cc:331] REPLICATION: starting
0806 20:16:02.949256 (+166571us) log.cc:842] Serialized 2470 byte log entry
0806 20:16:04.065724 (+1116468us) op_driver.cc:444] REPLICATION: finished
0806 20:16:04.576791 (+511067us) write_op.cc:301] APPLY: starting
0806 20:16:04.595093 (+ 18302us) tablet.cc:1365] starting BulkCheckPresence
0806 20:16:04.595304 (+ 211us) tablet.cc:1368] finished BulkCheckPresence
0806 20:16:04.595422 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:16:04.611402 (+ 15980us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:16:04.639280 (+ 27878us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0806 20:16:04.639331 (+ 51us) write_op.cc:312] APPLY: finished
0806 20:16:04.639600 (+ 269us) log.cc:842] Serialized 53 byte log entry
0806 20:16:04.639792 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:04.639906 (+ 114us) write_op.cc:454] Released schema lock
0806 20:16:04.639957 (+ 51us) write_op.cc:341] FINISH: Updating metrics
0806 20:16:04.640277 (+ 320us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:04.640303 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":188893,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":435843,"apply.run_cpu_time_us":3425,"apply.run_wall_time_us":100142,"num_ops":6,"prepare.queue_time_us":732388,"prepare.run_cpu_time_us":48742,"prepare.run_wall_time_us":1174844,"raft.queue_time_us":195642,"raft.run_cpu_time_us":32126,"raft.run_wall_time_us":421969,"replication_time_us":1263511,"spinlock_wait_cycles":689408,"thread_start_us":1281625,"threads_started":4,"wal-append.queue_time_us":386458,"wal-append.run_cpu_time_us":28780,"wal-append.run_wall_time_us":2288336}]]}
W20260806 20:16:05.785658 30757 leader_election.cc:334] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.592s (SENT)
I20260806 20:16:05.816648 30757 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:05.786815 748 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:05.788062 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1402) took 1326 ms. Trace:
I20260806 20:16:05.765265 32507 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:05.820199 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 82025us
I20260806 20:16:05.820649 30783 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:05.819118 30786 rpcz_store.cc:276] 0806 20:16:04.461856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.514708 (+ 52852us) service_pool.cc:227] Handling call
0806 20:16:05.787828 (+1273120us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b5400002b54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:05.787989 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2187957120}
I20260806 20:16:05.797951 809 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:05.789544 621 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:05.825062 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 50639.7us
W20260806 20:16:05.807799 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b060012888ff4d76b254b43d1f1959bb, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.826829 30834 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:05.828094 812 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:05.827384 30602 catalog_manager.cc:5495] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 2 to 3, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:05.828838 808 raft_consensus.cc:3032] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:05.790225 30867 rpcz_store.cc:276] 0806 20:16:02.948636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.049902 (+2101266us) service_pool.cc:227] Handling call
0806 20:16:05.050292 (+ 390us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:05.216974 (+166682us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:05.217040 (+ 66us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:05.369809 (+152769us) log.cc:842] Serialized 52 byte log entry
0806 20:16:05.548758 (+178949us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:05.577912 (+ 29154us) log.cc:842] Serialized 14 byte log entry
0806 20:16:05.578391 (+ 479us) log.cc:842] Serialized 14 byte log entry
0806 20:16:05.578566 (+ 175us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.590568 (+ 12002us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:05.689654 (+ 99086us) raft_consensus.cc:1671] finished
0806 20:16:05.689676 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.698276 (+ 8600us) inbound_call.cc:173] Queueing success response
0806 20:16:05.734645 (+ 36369us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b1800218560. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e753638 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d 00007f71579ab204 00007f71579ab039 00007f714e735dc6
Metrics: {"mutex_wait_us":179498,"prepare.queue_time_us":46467,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":73,"raft.queue_time_us":178909,"raft.run_cpu_time_us":463,"raft.run_wall_time_us":3931,"spinlock_wait_cycles":9020288,"thread_start_us":178479,"threads_started":1,"wal-append.queue_time_us":233498}
I20260806 20:16:05.801147 30870 rpcz_store.cc:276] 0806 20:16:02.952450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.052419 (+2099969us) service_pool.cc:227] Handling call
0806 20:16:05.052764 (+ 345us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:05.378784 (+326020us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:05.381963 (+ 3179us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:05.542388 (+160425us) log.cc:842] Serialized 26 byte log entry
0806 20:16:05.604772 (+ 62384us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:05.692390 (+ 87618us) log.cc:842] Serialized 14 byte log entry
0806 20:16:05.703862 (+ 11472us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.715773 (+ 11911us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:05.751367 (+ 35594us) raft_consensus.cc:1671] finished
0806 20:16:05.751387 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.751530 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":154458,"prepare.queue_time_us":85409,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":186099,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":7633,"thread_start_us":113642,"threads_started":2,"wal-append.queue_time_us":155055}
W20260806 20:16:05.829794 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 80312.5us
I20260806 20:16:05.841859 30835 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.838063 30756 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.844287 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1306) took 2790 ms. Trace:
I20260806 20:16:05.844764 30868 rpcz_store.cc:276] 0806 20:16:03.075761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:05.331041 (+2255280us) service_pool.cc:227] Handling call
0806 20:16:05.331534 (+ 493us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:05.692091 (+360557us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.838959 (+146868us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b540019d554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:05.839133 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":680000256}
I20260806 20:16:05.826007 31417 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:05.790158 682 consensus_queue.cc:237] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.823200 548 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:05.843430 32642 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 20:16:05.864292 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 138127us
I20260806 20:16:05.855561 30602 catalog_manager.cc:5495] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 5 to 6, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 6 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:16:05.841485 776 raft_consensus.cc:2726] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:16:05.860540 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 74961.4us
I20260806 20:16:05.866274 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:05.840081 809 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:05.841347 808 raft_consensus.cc:729] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:05.867877 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:16:05.854496 788 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:05.849050 812 raft_consensus.cc:686] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:05.866559 548 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:05.889050 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.869321 32642 raft_consensus.cc:686] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:05.887609 30710 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:05.887360 788 raft_consensus.cc:686] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:05.899106 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1293) took 2976 ms. Trace:
I20260806 20:16:05.900162 788 consensus_queue.cc:237] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:05.901803 30757 leader_election.cc:334] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.660s (SENT)
I20260806 20:16:05.902365 30757 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:05.902913 567 raft_consensus.cc:3032] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:05.903426 567 raft_consensus.cc:729] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:05.904623 800 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:05.904435 567 consensus_queue.cc:260] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.908394 30787 raft_consensus.cc:1226] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:05.826758 613 consensus_queue.cc:260] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.875955 753 consensus_queue.cc:1035] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20260806 20:16:05.909623 30602 catalog_manager.cc:5495] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 1 to 4, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 4 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
W20260806 20:16:05.904346 30757 leader_election.cc:334] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.963s (SENT)
I20260806 20:16:05.922585 32642 consensus_queue.cc:237] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.923331 613 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:05.926510 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 99839.1us
I20260806 20:16:05.927436 30834 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:05.929319 817 log.cc:924] Time spent T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.196s user 0.000s sys 0.002s
I20260806 20:16:05.891572 548 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:05.930612 30834 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:05.931205 30757 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:05.899189 807 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:05.934621 31389 leader_election.cc:334] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.554s (SENT)
W20260806 20:16:05.934530 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 55239.6us
I20260806 20:16:05.899642 30869 rpcz_store.cc:276] 0806 20:16:02.922614 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:04.996336 (+2073722us) service_pool.cc:227] Handling call
0806 20:16:05.015805 (+ 19469us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:05.387695 (+371890us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:05.394397 (+ 6702us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:05.650592 (+256195us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b2c001497e8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:16:05.651222 (+ 630us) log.cc:842] Serialized 52 byte log entry
0806 20:16:05.790569 (+139347us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:05.820882 (+ 30313us) log.cc:842] Serialized 14 byte log entry
0806 20:16:05.844339 (+ 23457us) log.cc:842] Serialized 14 byte log entry
0806 20:16:05.849266 (+ 4927us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:05.869362 (+ 20096us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:05.898714 (+ 29352us) raft_consensus.cc:1671] finished
0806 20:16:05.898739 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:05.898887 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":212262,"prepare.queue_time_us":41752,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":1456,"raft.queue_time_us":247727,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":10169,"spinlock_wait_cycles":270470912,"thread_start_us":190969,"threads_started":2,"wal-append.queue_time_us":244571}
I20260806 20:16:05.905246 800 raft_consensus.cc:686] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:16:05.943246 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 47387.2us
W20260806 20:16:05.944473 31389 leader_election.cc:334] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.649s (SENT)
I20260806 20:16:05.944927 31389 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:05.940729 351 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:05.947157 32733 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:05.947718 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:16:05.948123 608 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:05.948570 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:16:05.948247 618 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:05.956765 30676 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:05.941320 682 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:16:05.960698 30602 catalog_manager.cc:5495] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:16:05.952061 32733 raft_consensus.cc:686] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:16:05.966020 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 61722.7us
I20260806 20:16:05.917927 32647 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:05.948791 608 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.963665 567 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:05.958725 755 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:05.961330 599 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:16:05.980717 30757 leader_election.cc:334] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.998s (SENT)
W20260806 20:16:05.985948 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 56563.1us
I20260806 20:16:05.975503 619 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:16:05.988204 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 43779.7us
I20260806 20:16:05.992725 800 consensus_queue.cc:237] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.978984 823 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:05.997087 599 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.960597 618 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:05.999856 599 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:06.000895 30676 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.001514 618 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:06.011828 30710 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:06.013899 30834 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.017242 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:05.986658 32672 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:16:06.001766 753 raft_consensus.cc:2741] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:16:06.011847 30602 catalog_manager.cc:5495] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
W20260806 20:16:06.032232 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 55867.6us
I20260806 20:16:05.969480 808 consensus_queue.cc:260] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:05.988508 32733 consensus_queue.cc:237] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.041422 30710 raft_consensus.cc:1261] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:16:06.042219 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:06.035313 30602 catalog_manager.cc:5495] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 2 to 3, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 3 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:06.045879 32647 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:06.044677 30602 catalog_manager.cc:5495] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 1 to 5, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 5 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:16:06.050458 621 consensus_queue.cc:1035] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:06.051041 30602 catalog_manager.cc:5495] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:06.061228 668 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:06.061546 30602 catalog_manager.cc:5495] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 1 to 5, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 5 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } }
I20260806 20:16:05.988034 812 consensus_queue.cc:237] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.038093 32672 raft_consensus.cc:686] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:06.078039 32672 consensus_queue.cc:237] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.042363 810 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:16:06.084226 30602 catalog_manager.cc:5495] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 4 to 5, leader changed from 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 5 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:06.096668 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:06.097395 30791 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:06.099696 30757 leader_election.cc:334] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.631s (SENT)
I20260806 20:16:06.064182 809 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:06.101773 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 91264.8us
W20260806 20:16:06.100386 827 log.cc:924] Time spent T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260806 20:16:06.103528 31389 leader_election.cc:334] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.994s (SENT)
I20260806 20:16:06.103960 31389 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.111383 30757 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:06.110608 30791 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:06.035862 631 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.125720 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:06.090111 31417 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
W20260806 20:16:06.137310 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 150702us
I20260806 20:16:06.140327 31389 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:06.142445 30787 raft_consensus.cc:2379] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:06.130685 658 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:16:06.129958 808 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:06.136915 788 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 20:16:06.149639 826 log.cc:924] Time spent T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260806 20:16:06.150903 800 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:06.151453 800 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:06.152263 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.152905 30870 raft_consensus.cc:2362] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7. Re-sending same reply.
I20260806 20:16:06.144204 713 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:06.154332 713 raft_consensus.cc:686] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:06.155308 713 consensus_queue.cc:237] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:06.155836 800 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:06.158758 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:06.162287 809 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:06.163647 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.164526 30870 raft_consensus.cc:2379] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:06.167016 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:06.173581 30710 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:06.177490 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.187961 30602 catalog_manager.cc:5495] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 5 to 6, leader changed from 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 6 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:06.189144 631 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:06.189859 30602 catalog_manager.cc:5495] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 reported cstate change: term changed from 3 to 4, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 4 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:06.178175 787 consensus_queue.cc:1046] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260806 20:16:06.194375 787 raft_consensus.cc:3032] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:06.194751 787 raft_consensus.cc:729] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:06.195660 787 consensus_queue.cc:260] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:06.194952 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 95308.3us
I20260806 20:16:06.203897 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.204411 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.209179 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:06.209667 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.202934 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.204540 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.208707 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.204891 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.213630 30861 raft_consensus.cc:2379] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
W20260806 20:16:06.213776 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 672b273ef36e4c6eb6873cc61471cb53, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:06.214129 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 672b273ef36e4c6eb6873cc61471cb53, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.206588 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.216878 30865 raft_consensus.cc:2379] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:06.216981 608 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:06.197114 787 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:06.206889 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.219295 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.221375 30710 raft_consensus.cc:1226] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:06.210392 30868 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:16:06.202905 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.211024 30870 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:16:06.227068 30864 raft_consensus.cc:2362] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7. Re-sending same reply.
I20260806 20:16:06.221424 30676 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.177075 807 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:16:06.207254 30757 leader_election.cc:334] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.822s (SENT)
I20260806 20:16:06.233382 30757 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:06.234633 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.235783 631 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:16:06.247519 30867 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:06.249931 30871 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:16:06.242998 608 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:06.252401 30754 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:06.248939 631 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:06.262821 608 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:06.263465 608 raft_consensus.cc:686] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:06.264976 608 consensus_queue.cc:237] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:06.272397 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 65172.4us
I20260806 20:16:06.262030 631 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:16:06.281103 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 139896us
I20260806 20:16:06.201422 548 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:06.285848 30705 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:06.279774 668 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:06.292470 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:06.292366 668 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.294255 548 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:06.294513 668 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:06.295277 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:06.296037 31389 leader_election.cc:334] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.152s (SENT)
I20260806 20:16:06.296556 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.298362 30710 raft_consensus.cc:1261] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:16:06.299399 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:06.299974 667 consensus_queue.cc:1035] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:16:06.301805 30832 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.302608 618 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:06.303359 755 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:06.326267 30705 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:06.325724 32507 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:06.330039 32507 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.343833 30711 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:06.344483 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.343669 30709 raft_consensus.cc:1261] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:16:06.347250 667 consensus_queue.cc:1046] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:16:06.347891 668 consensus_queue.cc:1035] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:06.349633 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:06.352294 30787 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:06.353089 667 raft_consensus.cc:3032] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:06.353549 667 raft_consensus.cc:729] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:06.354419 667 consensus_queue.cc:260] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.355302 30835 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:06.355700 667 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:06.356616 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 71924.4us
W20260806 20:16:06.357504 837 log.cc:924] Time spent T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.145s user 0.002s sys 0.000s
I20260806 20:16:06.361528 32507 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:06.368714 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.371919 548 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:06.377143 30676 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.372493 32507 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 20:16:06.385275 30757 leader_election.cc:334] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.936s (SENT)
I20260806 20:16:06.383373 548 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:06.388918 548 raft_consensus.cc:686] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:06.389848 548 consensus_queue.cc:237] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.382416 32507 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:06.393303 32507 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:06.424520 32733 consensus_queue.cc:1046] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:16:06.424696 32507 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.426047 32672 raft_consensus.cc:3032] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:06.426478 32672 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:06.427366 32672 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:06.429606 32507 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:06.430158 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:06.430835 30787 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:06.438077 788 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:06.439473 788 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.447001 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.448827 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:06.449787 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.450055 30710 raft_consensus.cc:1226] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:06.451145 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.451831 30712 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:06.452699 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.449178 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.455080 30787 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:06.456312 30835 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:06.457120 32507 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:06.457540 30713 raft_consensus.cc:1261] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:16:06.458964 32672 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:06.462193 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:06.462790 30706 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:06.462934 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.463554 30711 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:06.464824 631 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.466020 30703 raft_consensus.cc:1226] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 2. Current term is 3. Ops: []
I20260806 20:16:06.466449 788 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:06.467078 631 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:06.469269 788 consensus_queue.cc:1046] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:16:06.470671 619 raft_consensus.cc:3032] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:16:06.471028 619 raft_consensus.cc:729] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:06.471590 30676 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.472044 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.471843 619 consensus_queue.cc:260] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.472715 32733 consensus_queue.cc:1035] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 20:16:06.473192 619 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:06.473328 30676 leader_election.cc:398] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6.
I20260806 20:16:06.476117 30676 leader_election.cc:401] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 20:16:06.472702 30703 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:06.478551 30756 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:06.475016 32733 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:16:06.482645 30787 raft_consensus.cc:1261] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:06.480149 831 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:06.457633 32507 raft_consensus.cc:686] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:06.493624 30703 raft_consensus.cc:3032] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:06.494098 30703 raft_consensus.cc:729] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:06.482230 788 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:06.485971 831 raft_consensus.cc:686] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 8 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:06.502154 30706 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:06.504590 631 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6.
I20260806 20:16:06.484611 548 consensus_queue.cc:1035] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.016s
I20260806 20:16:06.489575 755 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260806 20:16:06.495522 788 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:06.505863 748 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:06.522763 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:06.525843 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:06.529671 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:06.520396 808 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:06.536643 808 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:06.535369 30706 raft_consensus.cc:2379] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:06.518460 788 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:06.541280 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.541414 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.543071 30864 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:06.539695 621 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:06.544394 621 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:06.544095 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.546087 30869 raft_consensus.cc:2379] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:06.546711 621 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:06.547268 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.547410 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.547794 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:06.548467 31421 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:06.537644 32507 consensus_queue.cc:237] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.549046 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.549754 30870 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:06.551384 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.547410 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.552652 30871 raft_consensus.cc:2379] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:06.531180 748 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:06.556286 748 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:06.556835 30754 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:06.531076 578 consensus_queue.cc:1046] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260806 20:16:06.559195 30870 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:06.559366 753 raft_consensus.cc:3032] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:06.559717 753 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:06.560453 753 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.560959 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:06.562153 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.563789 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:06.558984 808 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:06.569403 30786 raft_consensus.cc:1261] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 20:16:06.570189 30871 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:06.570777 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:06.571012 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.573782 30864 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:06.572013 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.574740 30868 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:06.547920 30869 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:06.582289 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:16:06.583046 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 87658us
I20260806 20:16:06.584980 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:06.585002 30867 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:06.588207 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.592428 808 consensus_queue.cc:1035] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:16:06.597517 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.585873 621 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:06.598490 30835 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.576932 667 consensus_queue.cc:1035] T 6c8d5f5ccc46476aac7a641abae6bdfa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.018s
I20260806 20:16:06.601763 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.613524 30870 raft_consensus.cc:2379] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:06.608213 621 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:06.561748 831 consensus_queue.cc:237] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.519526 30703 consensus_queue.cc:260] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.615309 30871 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:06.714082 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.716773 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:06.722554 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 69109.4us
I20260806 20:16:06.714753 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.750249 30864 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:06.714710 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.751156 30870 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:06.732405 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 133379us
I20260806 20:16:06.615448 30868 raft_consensus.cc:1261] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 20:16:06.699879 30867 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:06.653033 30713 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:16:06.754237 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.757202 30863 raft_consensus.cc:2379] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:06.646688 808 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:16:06.648322 847 log.cc:924] Time spent T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20260806 20:16:06.615219 621 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:06.769625 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.701130 32672 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:06.717123 32507 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:06.775096 30867 raft_consensus.cc:2379] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:06.772399 578 consensus_queue.cc:1046] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260806 20:16:06.613653 548 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.157s
I20260806 20:16:06.774384 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.781540 32733 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:06.782166 613 consensus_queue.cc:1046] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:16:06.786223 30792 raft_consensus.cc:1261] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 20:16:06.796732 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:06.802841 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.803022 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.803108 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.803900 30871 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:06.760326 690 consensus_queue.cc:1046] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.041s
I20260806 20:16:06.767110 713 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260806 20:16:06.803570 30866 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:06.837924 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:06.808734 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.843766 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.844851 30785 raft_consensus.cc:1203] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 4.3->[6.4-6.4] Dedup: 6.4->[]
I20260806 20:16:06.848404 30871 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:06.809484 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:06.852296 30785 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260806 20:16:06.857939 30783 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:06.863306 753 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:06.850303 755 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:06.866114 755 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.870347 30783 raft_consensus.cc:1261] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:16:06.808460 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:06.807974 30870 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:06.885958 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.886444 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.887099 30868 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:06.886430 30865 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:06.893103 30791 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:06.896224 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.899194 30871 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:06.899936 30792 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:06.900357 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.900985 30871 raft_consensus.cc:2362] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5. Re-sending same reply.
I20260806 20:16:06.898475 30864 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
W20260806 20:16:06.896803 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 118997us
I20260806 20:16:06.905055 30791 raft_consensus.cc:1261] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:16:06.906553 838 consensus_queue.cc:1035] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:16:06.835942 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.785956 32672 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:06.922587 30864 raft_consensus.cc:1261] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 20:16:06.929335 30790 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:06.931912 30783 raft_consensus.cc:1261] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20260806 20:16:06.861182 788 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.877130 808 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:06.938169 857 consensus_queue.cc:1046] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:16:06.939394 30863 raft_consensus.cc:1261] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:16:06.950181 30868 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:06.940703 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.806228 32507 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.951036 30864 raft_consensus.cc:2379] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:06.941318 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.944712 31389 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.945672 30861 raft_consensus.cc:2379] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:06.938334 755 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:06.932320 30783 pending_rounds.cc:76] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Aborting all ops after (but not including) 2
I20260806 20:16:06.886705 812 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:06.952378 30871 raft_consensus.cc:2362] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4. Re-sending same reply.
I20260806 20:16:06.958673 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.959406 30868 raft_consensus.cc:2362] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6. Re-sending same reply.
I20260806 20:16:06.967561 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:06.806566 32733 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:06.957819 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:06.977115 30709 raft_consensus.cc:1261] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:16:06.973170 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.977834 30868 raft_consensus.cc:2362] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8. Re-sending same reply.
I20260806 20:16:06.974449 30712 raft_consensus.cc:1261] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:06.975127 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.975215 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:06.977263 30789 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:06.954394 30783 pending_rounds.cc:97] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260806 20:16:06.980211 30835 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:06.931061 690 consensus_queue.cc:1225] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Peer 58d59beb597f4ed99d84d102ddb41ff5 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 20:16:06.987095 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:06.987978 30868 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:06.979660 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:06.991242 30867 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:06.999972 30783 raft_consensus.cc:2866] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:06.992200 30864 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:06.997193 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:07.037206 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 92582.3us
W20260806 20:16:07.043637 30757 leader_election.cc:334] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.201s (SENT)
I20260806 20:16:06.987198 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:06.999763 810 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:07.064972 862 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 20:16:07.066133 859 log.cc:924] Time spent T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260806 20:16:07.066473 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:07.067153 30708 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:07.068383 30789 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 20:16:07.068620 621 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:07.070176 864 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:07.074013 32733 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:07.074381 30861 raft_consensus.cc:2362] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4. Re-sending same reply.
I20260806 20:16:07.074505 32672 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:07.078308 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:07.087165 30871 raft_consensus.cc:2362] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4. Re-sending same reply.
I20260806 20:16:07.037156 809 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.080657 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:06.981616 808 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.085316 30869 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:07.088028 31421 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
W20260806 20:16:07.076689 855 log.cc:924] Time spent T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20260806 20:16:07.094864 30869 raft_consensus.cc:1261] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:07.062145 810 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:07.102250 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:07.108429 30786 raft_consensus.cc:1261] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 20:16:07.113646 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 138523us
I20260806 20:16:07.084192 619 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.122133 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:07.133441 31389 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:07.134272 30871 raft_consensus.cc:2354] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6.
I20260806 20:16:07.013046 812 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:07.135965 30787 raft_consensus.cc:2396] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260806 20:16:07.141798 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 182020us
I20260806 20:16:07.143206 30834 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:07.101928 810 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:07.133244 30867 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:07.145504 30676 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:07.146538 30869 raft_consensus.cc:1261] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:16:07.152755 30871 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:07.161129 31389 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:07.161602 30787 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:07.166635 599 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.167173 599 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.167609 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:07.169092 599 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:07.170212 30786 raft_consensus.cc:1261] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:07.170373 506 raft_consensus.cc:3032] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:16:07.170523 30673 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:07.171849 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:07.096309 809 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:07.178328 30784 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:16:07.179287 30866 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
W20260806 20:16:07.198058 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 75870.8us
I20260806 20:16:07.198954 30757 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:16:07.207494 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 115235us
I20260806 20:16:07.211565 30788 raft_consensus.cc:1261] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:16:07.207803 787 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 20:16:07.261605 865 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:07.262143 865 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:07.142441 619 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.262522 865 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:07.256868 812 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:07.179710 506 raft_consensus.cc:729] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:07.243048 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:07.277284 32507 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:07.301944 599 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:07.302421 599 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:07.302801 599 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:07.286485 621 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:07.287931 810 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.291731 30783 raft_consensus.cc:1261] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260806 20:16:07.294857 874 log.cc:924] Time spent T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260806 20:16:07.290186 787 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.300702 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:07.301843 32507 consensus_queue.cc:1035] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260806 20:16:07.284358 875 log.cc:924] Time spent T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
W20260806 20:16:07.206871 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 53857.9us
I20260806 20:16:07.162317 548 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:16:07.317021 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 164647us
I20260806 20:16:07.173197 748 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.321480 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:07.200103 788 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:07.213501 823 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:07.335773 30676 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:07.323006 881 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:07.337383 690 consensus_queue.cc:1035] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:16:07.351799 506 consensus_queue.cc:260] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:07.353722 32733 consensus_queue.cc:1159] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" responder_term: 3 status { last_received { term: 3 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 3 index: 6 } }
I20260806 20:16:07.356310 619 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:07.353325 506 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:07.344478 881 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:07.357765 881 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:07.353770 690 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:07.361114 690 raft_consensus.cc:686] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 8 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:07.338358 873 consensus_queue.cc:1035] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:16:07.338806 30707 raft_consensus.cc:3032] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:07.339732 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:07.341564 810 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:07.343202 667 raft_consensus.cc:3032] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:07.320183 787 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:07.381533 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 255017us
I20260806 20:16:07.352034 808 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:07.364789 30707 raft_consensus.cc:729] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:07.351799 809 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:07.323159 548 raft_consensus.cc:686] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:07.382457 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:07.384908 30703 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:07.385643 872 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.386710 864 consensus_queue.cc:1035] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:16:07.387249 30711 raft_consensus.cc:1261] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 20:16:07.388019 30676 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:07.362123 690 consensus_queue.cc:237] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.417375 787 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:07.418483 30707 consensus_queue.cc:260] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.440929 30707 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:07.442030 30863 raft_consensus.cc:1226] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: []
I20260806 20:16:07.422991 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:07.317762 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:16:07.443113 32733 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:07.443701 32733 raft_consensus.cc:686] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:07.444491 32733 consensus_queue.cc:237] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.445957 30866 raft_consensus.cc:1261] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 20:16:07.447978 548 consensus_queue.cc:237] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.393998 667 raft_consensus.cc:729] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:07.449494 667 consensus_queue.cc:260] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.450667 667 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:07.440074 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 52141.7us
I20260806 20:16:07.457309 30676 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:07.461527 30703 raft_consensus.cc:1261] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260806 20:16:07.462841 30703 pending_rounds.cc:76] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 3
I20260806 20:16:07.463173 631 consensus_queue.cc:1035] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:07.463676 30676 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:07.440886 882 log.cc:924] Time spent T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260806 20:16:07.465719 599 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.348261 748 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.467069 30867 raft_consensus.cc:1261] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 20:16:07.471688 30703 pending_rounds.cc:97] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260806 20:16:07.473057 30707 raft_consensus.cc:1261] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260806 20:16:07.465493 32672 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:07.491317 30707 pending_rounds.cc:76] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 2
I20260806 20:16:07.491873 30707 pending_rounds.cc:97] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260806 20:16:07.492286 30707 raft_consensus.cc:2866] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 20:16:07.485172 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 102798us
I20260806 20:16:07.486052 30866 raft_consensus.cc:1261] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:16:07.439802 872 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:07.491043 30703 raft_consensus.cc:2866] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:07.499516 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:07.504849 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 380751us
I20260806 20:16:07.504922 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:07.505762 31421 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:07.508308 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:07.509126 30865 raft_consensus.cc:2379] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:07.509637 30832 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:16:07.510459 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 253438us
I20260806 20:16:07.512007 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:07.514665 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 51074.8us
W20260806 20:16:07.508136 578 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260806 20:16:07.523543 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:07.489826 871 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:07.526983 30713 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:16:07.529531 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:07.493889 32672 raft_consensus.cc:686] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:07.541566 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:07.515923 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1597) took 1037 ms. Trace:
I20260806 20:16:07.530751 30602 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41564 (request call id 228) took 5960 ms. Trace:
I20260806 20:16:07.514328 864 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:07.532562 30676 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:07.558873 31388 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:07.542587 30703 rpcz_store.cc:276] 0806 20:16:06.478361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:06.478474 (+ 113us) service_pool.cc:227] Handling call
0806 20:16:06.492228 (+ 13754us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:07.384405 (+892177us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54003c53a0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71515fd40d 00007f71515eb050 00007f715164eaba 00007f71516663af 00007f7151667b4e 00007f7151671c89 00007f7151672556 00007f715166eedf 00007f715166e99e 00007f71565b02e9
0806 20:16:07.515660 (+131255us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:07.515864 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":534542720}
I20260806 20:16:07.561359 30784 raft_consensus.cc:2379] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:07.545176 30791 raft_consensus.cc:2379] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:07.531546 883 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:07.549517 30603 catalog_manager.cc:5495] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:07.550398 599 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:07.528041 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:07.531649 810 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:16:07.565804 30603 catalog_manager.cc:5495] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 5 to 6, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 6 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:07.551853 748 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:07.552771 32507 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.553777 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:07.596362 30785 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:07.554435 808 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.557101 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:07.561751 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:07.597426 30786 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:07.534965 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:07.577636 30603 catalog_manager.cc:5495] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:07.599781 30787 raft_consensus.cc:2354] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8.
I20260806 20:16:07.543280 30602 rpcz_store.cc:276] 0806 20:16:01.566169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:01.566406 (+ 237us) service_pool.cc:227] Handling call
0806 20:16:07.526717 (+5960311us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:16:06.372283 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:16:06.394561 (+ 22278us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:16:06.394621 (+ 60us) write_op.cc:435] Acquired schema lock
0806 20:16:06.394637 (+ 16us) tablet.cc:660] Decoding operations
0806 20:16:06.396182 (+ 1545us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:16:06.396283 (+ 101us) write_op.cc:641] Partition lock acquired for write op
0806 20:16:06.396311 (+ 28us) tablet.cc:683] Acquiring locks for 17 operations
0806 20:16:06.452338 (+ 56027us) tablet.cc:699] Row locks acquired
0806 20:16:06.452383 (+ 45us) write_op.cc:260] PREPARE: finished
0806 20:16:06.452566 (+ 183us) write_op.cc:270] Start()
0806 20:16:06.452748 (+ 182us) write_op.cc:276] Timestamp: P: 1786047366452505 usec, L: 0
0806 20:16:06.452788 (+ 40us) op_driver.cc:331] REPLICATION: starting
0806 20:16:06.467526 (+ 14738us) log.cc:842] Serialized 6771 byte log entry
0806 20:16:06.765154 (+297628us) op_driver.cc:444] REPLICATION: finished
0806 20:16:06.896096 (+130942us) write_op.cc:301] APPLY: starting
0806 20:16:06.896210 (+ 114us) tablet.cc:1365] starting BulkCheckPresence
0806 20:16:06.896586 (+ 376us) tablet.cc:1368] finished BulkCheckPresence
0806 20:16:06.896734 (+ 148us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:16:06.898834 (+ 2100us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:16:06.899216 (+ 382us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0806 20:16:06.899260 (+ 44us) write_op.cc:312] APPLY: finished
0806 20:16:06.899732 (+ 472us) log.cc:842] Serialized 119 byte log entry
0806 20:16:06.900945 (+ 1213us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:06.901118 (+ 173us) write_op.cc:454] Released schema lock
0806 20:16:06.901214 (+ 96us) write_op.cc:341] FINISH: Updating metrics
0806 20:16:06.920352 (+ 19138us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:06.925971 (+ 5619us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":366877,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":121626,"apply.run_cpu_time_us":5172,"apply.run_wall_time_us":54169,"num_ops":17,"prepare.queue_time_us":54790,"prepare.run_cpu_time_us":42733,"prepare.run_wall_time_us":209524,"raft.queue_time_us":160151,"raft.run_cpu_time_us":10037,"raft.run_wall_time_us":139970,"replication_time_us":307268,"thread_start_us":297314,"threads_started":4,"wal-append.queue_time_us":106894}]]}
I20260806 20:16:07.597893 613 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:07.567781 618 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:07.610168 618 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:07.610548 618 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:07.612792 753 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.565055 883 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:07.613708 883 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:07.601656 32507 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.533257 30834 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:07.532853 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 269762us
I20260806 20:16:07.589478 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:07.613371 753 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.599095 30603 catalog_manager.cc:5495] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:16:07.563163 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 364286us
I20260806 20:16:07.539695 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:07.615769 30832 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:07.630692 30789 raft_consensus.cc:2379] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:07.630635 30603 catalog_manager.cc:5495] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 5, leader changed from 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 5 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:07.632470 30603 catalog_manager.cc:5495] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:07.638962 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:07.644384 30603 catalog_manager.cc:5495] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:07.614136 864 raft_consensus.cc:686] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:07.532336 881 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:07.622821 506 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:07.577520 852 consensus_queue.cc:1035] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.084s
I20260806 20:16:07.697990 30757 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:07.646271 30603 catalog_manager.cc:5495] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:07.702133 30603 catalog_manager.cc:5495] T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 5, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 5 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:07.614182 808 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.731257 30603 catalog_manager.cc:5495] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: config changed from index 2 to 5, term changed from 4 to 6, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to <none>, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New cstate: current_term: 6 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:07.748289 30603 catalog_manager.cc:5495] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 2 to 4, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 4 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:07.683444 810 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:16:07.729646 807 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.015s sys 0.000s
W20260806 20:16:07.729332 831 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.875s user 0.006s sys 0.009s
I20260806 20:16:07.637887 32672 consensus_queue.cc:237] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:07.779284 30603 catalog_manager.cc:5495] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:07.682636 506 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:07.990471 30868 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.019s sys 0.022s
I20260806 20:16:08.125658 30866 raft_consensus.cc:1203] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Deduplicated request from leader. Original: 3.2->[5.3-5.3] Dedup: 5.3->[]
W20260806 20:16:08.031215 30711 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.016s sys 0.008s
I20260806 20:16:08.126588 30866 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260806 20:16:08.126756 30711 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.662s user 0.016s sys 0.012s
W20260806 20:16:08.125872 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 494397us
W20260806 20:16:08.128259 30868 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s user 0.021s sys 0.027s
I20260806 20:16:08.129500 621 consensus_queue.cc:1035] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 20:16:08.023715 30785 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:08.132498 30757 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:16:08.125730 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 556731us
I20260806 20:16:08.134224 30603 catalog_manager.cc:5495] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 2 to 4, leader changed from 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 4 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:16:08.125419 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 592213us
I20260806 20:16:07.798919 548 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.163596 30757 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:07.791014 866 log.cc:924] Time spent T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
W20260806 20:16:07.790103 831 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.936s user 0.007s sys 0.010s
I20260806 20:16:08.169001 30835 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:07.790149 807 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.646s user 0.015s sys 0.000s
W20260806 20:16:08.024678 667 consensus_meta.cc:287] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.574s user 0.006s sys 0.003s
I20260806 20:16:08.167049 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1474) took 1278 ms. Trace:
I20260806 20:16:07.761370 888 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:08.179064 30835 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:07.755823 865 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:08.075605 613 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:08.179930 30870 rpcz_store.cc:276] 0806 20:16:06.867082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:06.926611 (+ 59529us) service_pool.cc:227] Handling call
0806 20:16:06.940147 (+ 13536us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:06.991286 (+ 51139us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:06.991317 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:07.341363 (+350046us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b2c001497e8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:16:07.341893 (+ 530us) log.cc:842] Serialized 52 byte log entry
0806 20:16:07.729924 (+388031us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:16:08.144351 (+414427us) log.cc:842] Serialized 14 byte log entry
0806 20:16:08.144621 (+ 270us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:08.144707 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:08.144750 (+ 43us) raft_consensus.cc:1671] finished
0806 20:16:08.144836 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:08.144981 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18079,"prepare.queue_time_us":432080,"raft.queue_time_us":40307,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":24229,"spinlock_wait_cycles":27635328,"thread_start_us":387642,"threads_started":1,"wal-append.queue_time_us":789274}
I20260806 20:16:08.097589 881 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:08.185811 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1476) took 1292 ms. Trace:
I20260806 20:16:07.867478 852 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.084228 753 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:07.848050 872 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:08.172461 668 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:08.172683 621 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:08.200874 30868 rpcz_store.cc:276] 0806 20:16:06.867791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:06.988610 (+120819us) service_pool.cc:227] Handling call
0806 20:16:07.397045 (+408435us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:16:07.423701 (+ 26656us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:07.461893 (+ 38192us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:16:07.466421 (+ 4528us) log.cc:842] Serialized 14 byte log entry
0806 20:16:08.129299 (+662878us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:08.129352 (+ 53us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:08.129402 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:08.160094 (+ 30692us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:08.160270 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4921,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1694,"raft.queue_time_us":110,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":104,"spinlock_wait_cycles":31739776,"thread_start_us":661957,"threads_started":1}
I20260806 20:16:08.181342 879 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 20:16:08.226765 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 63262.5us
I20260806 20:16:08.162088 809 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:08.178828 857 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.187367 30676 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:08.270403 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 54261.9us
I20260806 20:16:08.189566 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:08.267957 30712 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:08.273963 30792 raft_consensus.cc:2379] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:08.216759 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:08.219252 32507 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:08.199741 619 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.178696 631 consensus_queue.cc:1046] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260806 20:16:08.292089 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:08.293017 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:08.277624 30757 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:16:08.279157 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 95355.3us
W20260806 20:16:08.273329 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 86047.7us
I20260806 20:16:08.299527 30676 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:08.199718 667 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:08.280643 871 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.282344 862 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.262959 879 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:08.284390 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:08.284547 808 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:07.853155 864 consensus_queue.cc:237] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:08.283463 618 consensus_meta.cc:287] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.672s user 0.010s sys 0.000s
I20260806 20:16:08.203804 877 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:08.309799 753 consensus_queue.cc:1035] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 20:16:08.304581 844 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.199497 690 consensus_queue.cc:1035] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.100s
W20260806 20:16:08.295552 894 log.cc:924] Time spent T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260806 20:16:08.283562 506 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:08.309607 831 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:16:08.321481 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:08.332449 30676 leader_election.cc:398] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8.
I20260806 20:16:08.178587 548 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:08.265555 885 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:08.350329 668 raft_consensus.cc:686] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:08.293823 578 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:16:08.206430 613 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:08.401753 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 227618us
I20260806 20:16:08.215660 852 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.413940 30783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:08.314451 865 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:08.356302 31416 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:08.418689 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 86556.4us
I20260806 20:16:08.356760 808 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.306903 619 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.432009 30783 raft_consensus.cc:595] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
W20260806 20:16:08.435243 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 157690us
I20260806 20:16:08.276682 869 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.413553 877 raft_consensus.cc:686] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:08.356822 618 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:08.326385 872 raft_consensus.cc:3032] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:08.291244 857 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.413285 807 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.508720 30792 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 8. Current term is 9. Ops: []
W20260806 20:16:08.404100 713 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.014s sys 0.022s
I20260806 20:16:08.325840 788 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:16:08.510568 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 206164us
I20260806 20:16:08.312867 667 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.557705 30866 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:08.328495 879 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:08.435437 831 raft_consensus.cc:686] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:08.584851 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:08.586311 30870 raft_consensus.cc:1261] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
W20260806 20:16:08.595610 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 69991.9us
I20260806 20:16:08.595758 30868 raft_consensus.cc:1261] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 20:16:08.512564 823 consensus_meta.cc:287] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.164s user 0.012s sys 0.001s
I20260806 20:16:08.596345 30757 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:08.597010 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:08.597886 30783 raft_consensus.cc:2354] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8.
I20260806 20:16:08.589749 30784 raft_consensus.cc:1261] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
W20260806 20:16:08.538995 883 consensus_meta.cc:287] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.925s user 0.009s sys 0.004s
I20260806 20:16:08.593763 30865 raft_consensus.cc:1261] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:08.592109 30871 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 20:16:08.602849 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:08.563710 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:08.538267 755 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.327122 871 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.594374 30792 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:08.603825 30787 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
W20260806 20:16:08.685397 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 253873us
I20260806 20:16:08.599161 30784 pending_rounds.cc:76] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Aborting all ops after (but not including) 4
I20260806 20:16:08.600639 668 consensus_queue.cc:237] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.594329 30834 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:08.588795 713 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.803s user 0.015s sys 0.023s
I20260806 20:16:08.594620 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:08.701994 30866 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:08.326948 862 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:08.538573 889 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.633867 823 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:08.706700 31421 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:08.563424 810 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.582108 872 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:08.693141 30868 raft_consensus.cc:1261] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:16:08.444728 506 consensus_queue.cc:1035] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.142s
I20260806 20:16:08.697048 30784 pending_rounds.cc:97] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260806 20:16:08.697654 30789 raft_consensus.cc:2396] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 20:16:08.591588 892 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.732996 892 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.413370 844 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:08.590121 788 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:08.545547 808 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.725780 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:08.707906 31417 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 8: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:08.717918 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:08.703388 898 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:08.729101 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 141555us
I20260806 20:16:08.693559 608 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:08.761605 30834 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:08.731024 30784 raft_consensus.cc:2866] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:08.739794 31419 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:16:08.751749 30787 raft_consensus.cc:595] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:16:08.704213 32507 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:08.763343 30787 raft_consensus.cc:603] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:16:08.593279 807 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.564215 869 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.713747 838 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:08.751089 865 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:08.754357 788 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:08.682968 883 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:08.822054 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 109374us
I20260806 20:16:08.698493 755 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:08.806480 32507 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.730996 810 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:08.814026 809 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:08.966481 30863 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 8: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:08.847653 690 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.969262 690 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:08.997825 928 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:08.738157 877 consensus_queue.cc:237] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:08.998100 900 log.cc:924] Time spent T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.474s user 0.001s sys 0.001s
I20260806 20:16:08.732712 831 consensus_queue.cc:237] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:08.997692 903 log.cc:924] Time spent T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.394s user 0.000s sys 0.001s
I20260806 20:16:08.713613 889 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:08.969619 908 log.cc:924] Time spent T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20260806 20:16:08.965946 32672 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:08.785919 608 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.074357 931 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:08.966202 809 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:16:09.133940 902 log.cc:924] Time spent T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.108s user 0.000s sys 0.004s
I20260806 20:16:09.137921 755 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:09.130506 910 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:09.218489 30791 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 20:16:08.890885 865 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:09.204843 30712 consensus_meta.cc:287] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.931s user 0.008s sys 0.004s
I20260806 20:16:09.243566 30712 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:08.785835 898 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:09.225414 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 259659us
I20260806 20:16:09.227905 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:08.950387 32507 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:09.237928 31389 leader_election.cc:334] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.856s (SENT)
I20260806 20:16:09.066797 32733 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.050411 878 consensus_queue.cc:1035] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.638s
W20260806 20:16:09.224902 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 463364us
I20260806 20:16:08.814857 838 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.103890 618 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:09.141021 30673 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:08.851627 872 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:09.417660 30783 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.001s sys 0.005s
W20260806 20:16:09.254987 864 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.000s sys 0.014s
W20260806 20:16:09.438366 907 log.cc:924] Time spent T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.682s user 0.000s sys 0.001s
W20260806 20:16:09.677569 864 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.075s user 0.006s sys 0.034s
W20260806 20:16:09.307915 30756 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.001s sys 0.013s
W20260806 20:16:09.319630 30757 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.003s sys 0.014s
W20260806 20:16:09.610781 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 469839us
W20260806 20:16:09.513135 30675 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.008s sys 0.022s
W20260806 20:16:09.680719 30757 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.944s user 0.009s sys 0.032s
W20260806 20:16:09.680757 30675 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.944s user 0.010s sys 0.023s
W20260806 20:16:09.681532 30783 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.871s user 0.001s sys 0.015s
W20260806 20:16:09.684082 30756 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.981s user 0.007s sys 0.016s
W20260806 20:16:09.685056 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 1.06922e+06us
I20260806 20:16:09.244216 32672 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.238220 809 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:09.244215 578 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.005s sys 0.024s
W20260806 20:16:09.682336 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 386273us
I20260806 20:16:09.690346 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1479) took 2704 ms. Trace:
W20260806 20:16:09.609265 788 consensus_meta.cc:287] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.720s user 0.007s sys 0.000s
I20260806 20:16:09.691620 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:09.555962 933 log.cc:924] Time spent T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20260806 20:16:09.562067 915 log.cc:924] Time spent T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.627s user 0.001s sys 0.000s
W20260806 20:16:09.254405 812 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.964s user 0.004s sys 0.008s
W20260806 20:16:09.411847 903 log.cc:924] Time spent T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260806 20:16:09.574859 898 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:09.562331 909 log.cc:924] Time spent T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.696s user 0.000s sys 0.001s
I20260806 20:16:09.694554 810 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:09.554335 917 log.cc:924] Time spent T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
W20260806 20:16:09.363076 913 log.cc:924] Time spent T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.566s user 0.001s sys 0.001s
I20260806 20:16:08.888273 844 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:09.548986 31416 consensus_meta.cc:287] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.991s user 0.009s sys 0.000s
I20260806 20:16:09.696908 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:09.771771 30787 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
W20260806 20:16:09.567070 899 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260806 20:16:09.678407 864 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:09.602748 935 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.438596 936 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.699304 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:09.702243 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 464338us
I20260806 20:16:09.702657 31421 raft_consensus.cc:2396] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260806 20:16:09.404672 883 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:09.615983 878 consensus_queue.cc:1225] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer 42cabca48db7403b8523a3382a046043 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:16:09.761699 608 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:09.342540 862 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:09.286660 755 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.266248 885 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:09.684316 30757 connection.cc:232] RPC call timeout handler was delayed by -1.2109287140192464s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
I20260806 20:16:09.328843 506 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.696820 30861 rpcz_store.cc:276] 0806 20:16:06.974297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:07.121977 (+147680us) service_pool.cc:227] Handling call
0806 20:16:07.155211 (+ 33234us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:07.514219 (+359008us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:07.514251 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:07.527640 (+ 13389us) log.cc:842] Serialized 26 byte log entry
0806 20:16:07.861693 (+334053us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:08.182657 (+320964us) log.cc:842] Serialized 14 byte log entry
0806 20:16:08.182878 (+ 221us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:08.182952 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:09.678721 (+1495769us) raft_consensus.cc:1671] finished
0806 20:16:09.678741 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:09.678867 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":300465,"prepare.queue_time_us":420230,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":149027,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":16195,"thread_start_us":616534,"threads_started":2,"wal-append.queue_time_us":722312}
I20260806 20:16:09.456913 548 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:09.314731 631 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:09.884918 631 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:09.565855 889 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:09.710664 30785 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.003s sys 0.018s
I20260806 20:16:09.887005 883 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:09.887368 30785 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.997s user 0.003s sys 0.018s
I20260806 20:16:09.887575 883 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:09.676990 852 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:09.433030 807 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:09.888944 807 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.889398 807 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: NON_VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: true } }
I20260806 20:16:09.889909 883 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:09.891546 807 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:09.416114 884 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:09.423763 808 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:09.892238 884 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.407518 869 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:09.714651 30834 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:09.348241 892 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:09.715922 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:09.701481 578 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.984s user 0.011s sys 0.046s
I20260806 20:16:09.894423 30713 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:09.555150 927 log.cc:924] Time spent T e1485b55849242feae0e8b2b3b28307f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20260806 20:16:09.713840 30754 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.006s sys 0.014s
W20260806 20:16:09.898375 30754 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.195s user 0.006s sys 0.014s
W20260806 20:16:09.709072 753 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.008s sys 0.018s
I20260806 20:16:09.766398 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:09.712702 812 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.422s user 0.017s sys 0.024s
I20260806 20:16:09.678861 32733 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:09.712320 621 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:09.775123 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1477) took 2907 ms. Trace:
W20260806 20:16:09.722673 928 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.006s sys 0.016s
I20260806 20:16:09.908373 613 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:09.908766 928 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.910s user 0.007s sys 0.017s
I20260806 20:16:09.909057 928 raft_consensus.cc:686] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:09.724588 871 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:09.778651 30707 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:09.705582 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1815) took 1347 ms. Trace:
I20260806 20:16:09.710479 788 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:09.760453 898 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:09.943809 898 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:09.946142 898 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:09.253930 910 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.948241 910 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:09.786487 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1436) took 1276 ms. Trace:
I20260806 20:16:09.787842 31419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 491) took 1488 ms. Trace:
I20260806 20:16:09.788491 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:09.790110 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:09.773710 864 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:09.245085 931 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:09.776849 865 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:09.861524 31387 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.860s user 0.003s sys 0.018s
I20260806 20:16:09.950260 864 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:09.950572 31387 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.213s user 0.004s sys 0.018s
I20260806 20:16:09.952250 931 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:09.952327 920 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:09.948817 30790 rpcz_store.cc:276] 0806 20:16:08.510575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.510728 (+ 153us) service_pool.cc:227] Handling call
0806 20:16:08.563571 (+ 52843us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:08.564005 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:08.564034 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:08.591757 (+ 27723us) log.cc:842] Serialized 26 byte log entry
0806 20:16:08.727474 (+135717us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:16:08.727531 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:08.727615 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:09.786286 (+1058671us) raft_consensus.cc:1671] finished
0806 20:16:09.786307 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:09.786423 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3074,"raft.queue_time_us":1152,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":7583,"thread_start_us":32255,"threads_started":1,"wal-append.queue_time_us":765003}
I20260806 20:16:09.955854 30787 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:09.956420 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:09.956938 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:09.953066 920 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.960043 920 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:09.773924 31416 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
W20260806 20:16:09.790822 31389 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.938s (SENT) [suppressed 18 similar messages]
W20260806 20:16:09.961103 31416 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 490) took 1690 ms (client timeout 1838 ms). Trace:
W20260806 20:16:09.961884 31416 rpcz_store.cc:269] 0806 20:16:08.271041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.284073 (+ 13032us) service_pool.cc:227] Handling call
0806 20:16:09.961037 (+1676964us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:09.894557 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 179969us
I20260806 20:16:09.963961 30834 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:09.785578 618 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:16:09.898701 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 1.27833e+06us
W20260806 20:16:09.861950 879 consensus_meta.cc:287] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.166s user 0.009s sys 0.001s
I20260806 20:16:09.966912 879 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:09.900785 31386 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.137s user 0.017s sys 0.007s
W20260806 20:16:09.967871 31386 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.409s user 0.017s sys 0.007s
I20260806 20:16:09.967343 864 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:09.901834 30674 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.917s user 0.015s sys 0.006s
W20260806 20:16:09.969154 30674 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.170s user 0.015s sys 0.006s
W20260806 20:16:09.902390 934 log.cc:924] Time spent T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.266s user 0.001s sys 0.000s
I20260806 20:16:09.908118 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:09.970999 30783 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
W20260806 20:16:09.883469 909 log.cc:924] Time spent T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260806 20:16:09.863003 844 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:09.906487 30864 rpcz_store.cc:276] 0806 20:16:06.868100 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:07.020059 (+151959us) service_pool.cc:227] Handling call
0806 20:16:07.078335 (+ 58276us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:07.494233 (+415898us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:07.494261 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:07.508080 (+ 13819us) log.cc:842] Serialized 52 byte log entry
0806 20:16:07.924145 (+416065us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:08.211889 (+287744us) log.cc:842] Serialized 14 byte log entry
0806 20:16:08.262518 (+ 50629us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:08.262610 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:09.774910 (+1512300us) raft_consensus.cc:1671] finished
0806 20:16:09.774926 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:09.775065 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":266482,"prepare.queue_time_us":358019,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":247299,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":15772,"spinlock_wait_cycles":34304,"thread_start_us":773326,"threads_started":2,"wal-append.queue_time_us":889736}
I20260806 20:16:09.910187 928 consensus_queue.cc:237] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.969872 862 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:09.974248 862 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.976544 862 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:09.788005 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1439) took 1258 ms. Trace:
I20260806 20:16:09.972576 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1486) took 2461 ms. Trace:
I20260806 20:16:09.894846 808 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:09.977242 30867 rpcz_store.cc:276] 0806 20:16:07.510953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:07.511064 (+ 111us) service_pool.cc:227] Handling call
0806 20:16:07.511372 (+ 308us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:07.521725 (+ 10353us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:07.521752 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:07.540884 (+ 19132us) log.cc:842] Serialized 26 byte log entry
0806 20:16:08.054978 (+514094us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:08.440487 (+385509us) log.cc:842] Serialized 14 byte log entry
0806 20:16:08.504282 (+ 63795us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:08.504832 (+ 550us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:09.972363 (+1467531us) raft_consensus.cc:1671] finished
0806 20:16:09.972385 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:09.972516 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":169786,"prepare.queue_time_us":227378,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":124635,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":25446,"thread_start_us":596941,"threads_started":2,"wal-append.queue_time_us":1016819}
I20260806 20:16:09.973076 844 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.979934 944 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.980549 944 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.949649 31419 rpcz_store.cc:276] 0806 20:16:08.299349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.302458 (+ 3109us) service_pool.cc:227] Handling call
0806 20:16:08.302765 (+ 307us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:08.762917 (+460152us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:09.787610 (+1024693us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b54002cecd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:09.787781 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":983810304}
I20260806 20:16:09.964825 618 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.986155 31421 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:16:09.989126 879 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:09.989840 30869 raft_consensus.cc:2379] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 8: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:09.977070 30791 rpcz_store.cc:276] 0806 20:16:08.529099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.529211 (+ 112us) service_pool.cc:227] Handling call
0806 20:16:08.529520 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:09.236291 (+706771us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:09.787797 (+551506us) spinlock_profiling.cc:243] Waited 689 ms on lock 0x7b54001d1fd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:09.787951 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1583629696}
I20260806 20:16:10.001303 618 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:09.902904 753 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.937s user 0.012s sys 0.022s
I20260806 20:16:09.957198 30790 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:16:09.999388 944 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:09.887054 631 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:10.008821 30757 leader_election.cc:334] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.574s (SENT)
W20260806 20:16:10.009327 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 69777.5us
W20260806 20:16:10.010281 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 46394.2us
I20260806 20:16:09.973228 864 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:09.926234 869 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.929618 30788 rpcz_store.cc:276] 0806 20:16:08.358633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.358866 (+ 233us) service_pool.cc:227] Handling call
0806 20:16:08.422543 (+ 63677us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:08.691019 (+268476us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:16:08.691390 (+ 371us) log.cc:842] Serialized 14 byte log entry
0806 20:16:08.751897 (+ 60507us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:08.751958 (+ 61us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:08.752010 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:09.705282 (+953272us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b5400260554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:09.705336 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:09.705512 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13114,"prepare.queue_time_us":4132,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":624312,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":103,"spinlock_wait_cycles":599562112,"thread_start_us":47007,"threads_started":1,"wal-append.queue_time_us":729992}
I20260806 20:16:09.823427 755 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:09.911206 889 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:09.780807 935 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:09.782953 936 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.056160 30834 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:10.059180 947 log.cc:924] Time spent T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20260806 20:16:10.059643 922 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.060204 953 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:10.060787 808 consensus_queue.cc:1035] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:10.057389 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.045962 807 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:16:10.053237 879 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.903193 506 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.050904 892 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:10.062793 32507 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:09.926596 621 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.057551 548 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:09.900102 885 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:10.091346 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 300531us
I20260806 20:16:10.064626 812 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.063894 883 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:16:10.063596 31417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49046 (request call id 585) took 1872 ms (client timeout 2203 ms). Trace:
I20260806 20:16:10.101051 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.074450 32672 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.102408 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:10.103121 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 94341us
I20260806 20:16:10.110311 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.076483 631 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:10.066412 838 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.111958 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.113729 30703 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 8.
I20260806 20:16:10.111184 30706 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:16:10.119460 31388 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:10.119833 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.121666 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.091437 884 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.123960 30791 raft_consensus.cc:2379] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:10.099162 807 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 20:16:10.124168 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 46879.3us
I20260806 20:16:10.062674 869 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:10.103829 31417 rpcz_store.cc:269] 0806 20:16:08.191465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.206410 (+ 14945us) service_pool.cc:227] Handling call
0806 20:16:10.061165 (+1854755us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b54004af954. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:10.063513 (+ 2348us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":717486848}
I20260806 20:16:10.138135 883 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:10.139489 940 log.cc:924] Time spent T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.368s user 0.000s sys 0.001s
I20260806 20:16:10.140762 883 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.141587 948 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:10.142175 788 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.142319 948 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.143273 961 consensus_queue.cc:1035] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260806 20:16:10.088649 953 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.066237 889 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.824186 937 consensus_queue.cc:1035] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.625s
I20260806 20:16:10.148360 952 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.095949 879 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.148944 952 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.150396 935 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:10.150935 879 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:10.151117 952 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.121186 631 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.153685 631 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.153999 889 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.155159 30792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 252) took 1454 ms. Trace:
I20260806 20:16:10.155536 30792 rpcz_store.cc:276] 0806 20:16:08.700816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.706842 (+ 6026us) service_pool.cc:227] Handling call
0806 20:16:08.707079 (+ 237us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:09.305926 (+598847us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:09.305962 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:09.306007 (+ 45us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:09.306065 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:10.154922 (+848857us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b5400000d54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:10.154969 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:10.155098 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1376662272}
I20260806 20:16:10.074151 871 consensus_queue.cc:1035] T ec226f72b38a49529b88339beb36af46 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260806 20:16:10.101069 892 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:09.904922 945 consensus_queue.cc:1035] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.162s
I20260806 20:16:10.158186 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1491) took 1614 ms. Trace:
I20260806 20:16:10.158598 30863 rpcz_store.cc:276] 0806 20:16:08.543837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.543945 (+ 108us) service_pool.cc:227] Handling call
0806 20:16:10.157961 (+1614016us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b54002ba7d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:10.158095 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":831723264}
I20260806 20:16:10.159116 892 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.110540 548 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:10.138789 958 log.cc:924] Time spent T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260806 20:16:10.148038 755 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:10.137810 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:10.161423 548 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:10.162242 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 99694.6us
I20260806 20:16:10.163182 31388 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:10.147984 953 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.164904 31419 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 8: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:10.163367 30707 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:16:10.137955 943 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.168584 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.170879 31419 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:16:10.115769 942 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:10.085718 922 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.147294 32733 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.128029 30713 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 8.
I20260806 20:16:10.111063 32507 consensus_queue.cc:1035] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
W20260806 20:16:10.148139 949 log.cc:924] Time spent T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.248s user 0.000s sys 0.001s
I20260806 20:16:10.133462 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.160991 885 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:10.187095 506 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.173286 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.190490 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.191385 30710 raft_consensus.cc:2396] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 7 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260806 20:16:10.192255 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.192788 922 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.173599 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.194433 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:10.135439 618 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:10.194763 869 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.138070 938 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:10.196444 938 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.198650 938 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:10.201905 30712 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1461) took 2667 ms (client timeout 1838 ms). Trace:
I20260806 20:16:10.176520 871 consensus_queue.cc:1035] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
W20260806 20:16:10.202379 30712 rpcz_store.cc:269] 0806 20:16:07.534539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:07.534710 (+ 171us) service_pool.cc:227] Handling call
0806 20:16:10.201667 (+2666957us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b54001b4254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:10.201797 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1447267072}
I20260806 20:16:10.176499 943 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.203742 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.204643 30712 raft_consensus.cc:2396] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 20:16:10.208096 872 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:10.211373 809 consensus_meta.cc:287] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.507s user 0.005s sys 0.005s
I20260806 20:16:10.174998 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:10.175132 30673 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:10.175313 30835 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.175933 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:10.253162 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 78097.7us
I20260806 20:16:10.178799 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.255095 31419 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:16:10.181699 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.141582 936 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.180707 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.183158 31388 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:10.219194 823 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:10.222028 619 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.124045 812 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.177718 954 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:10.180780 942 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.230494 621 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.230960 667 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.231566 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1492) took 1669 ms. Trace:
I20260806 20:16:10.233268 921 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:10.134845 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 435417us
I20260806 20:16:10.234220 30706 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 8: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:10.234596 844 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.195138 618 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:16:10.236768 950 log.cc:924] Time spent T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
I20260806 20:16:10.205291 943 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:10.276568 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 101335us
I20260806 20:16:10.277572 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.184644 948 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.241348 873 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:10.248708 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.193280 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.282140 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.256069 30709 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:10.260502 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:10.259238 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 76183.3us
I20260806 20:16:10.260720 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:10.265482 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 152283us
I20260806 20:16:10.305770 30708 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
W20260806 20:16:10.306392 30757 leader_election.cc:334] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.799s (SENT)
W20260806 20:16:10.266412 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 214276us
I20260806 20:16:10.307144 30754 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.267578 30869 rpcz_store.cc:276] 0806 20:16:08.562702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.581300 (+ 18598us) service_pool.cc:227] Handling call
0806 20:16:10.231330 (+1650030us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54001e4454. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:10.231488 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3187907456}
I20260806 20:16:10.269361 30676 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:10.304080 30712 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:10.306550 31388 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:10.275231 621 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:10.262547 809 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.314845 914 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:10.315313 914 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:10.315357 809 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.263120 862 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:10.315755 914 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:10.275434 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.317287 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.317495 621 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.267552 619 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.318486 888 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.318876 619 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.274384 618 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:10.176460 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.341857 30705 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:10.342906 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.343008 30832 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:10.306669 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.311394 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.269521 936 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:10.275607 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.303578 935 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:10.383334 690 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.321103 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:10.263731 954 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:16:10.337261 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 59759.6us
I20260806 20:16:10.384064 30712 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:10.384893 30835 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.388305 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.389209 30787 raft_consensus.cc:2396] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 20:16:10.389127 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.390061 30792 raft_consensus.cc:2379] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:10.390295 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.337565 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.391453 30709 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:10.303467 667 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:10.393850 667 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.339277 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.395817 667 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:10.340443 30757 leader_election.cc:334] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.910s (SENT)
I20260806 20:16:10.318426 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.348927 972 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.355154 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.397518 30708 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:10.356989 942 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.358366 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.358748 30784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:10.400058 30705 raft_consensus.cc:2396] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 20:16:10.400285 30784 raft_consensus.cc:595] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:16:10.396763 30757 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:10.397773 954 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:10.359503 969 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260806 20:16:10.361091 857 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.415723 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.415951 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.312443 873 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.363130 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.363657 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.341748 871 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:10.365084 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.368911 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:10.369793 30832 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:10.370229 810 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:16:10.370442 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 49465.1us
I20260806 20:16:10.343971 608 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:10.440263 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.276377 922 consensus_queue.cc:1035] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20260806 20:16:10.334936 862 raft_consensus.cc:686] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:10.376897 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53732 (request call id 254) took 1152 ms. Trace:
I20260806 20:16:10.445529 30789 rpcz_store.cc:276] 0806 20:16:09.224896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:09.227571 (+ 2675us) service_pool.cc:227] Handling call
0806 20:16:10.376685 (+1149114us) spinlock_profiling.cc:243] Waited 657 ms on lock 0x7b54003b29d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:10.376830 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1510529280}
I20260806 20:16:10.276237 506 consensus_queue.cc:1035] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.066s
I20260806 20:16:10.446609 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.446774 753 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:16:10.448274 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.344677 944 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:10.448904 944 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:10.379204 936 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.380815 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.344179 831 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:10.382715 619 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.383134 618 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.383672 935 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:10.390830 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.398065 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.346318 809 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:10.359148 621 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.406785 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.396720 30712 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:10.408255 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.382766 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1435) took 1874 ms. Trace:
I20260806 20:16:10.409938 942 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:16:10.411639 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.412514 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:10.413611 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:10.416687 30705 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:10.399874 972 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.402679 879 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 20:16:10.422015 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 110712us
I20260806 20:16:10.370393 812 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:10.426487 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:10.428601 30655 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet 5ba1969a3911423e8e51fc4f2962515d
W20260806 20:16:10.434157 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 64439.6us
I20260806 20:16:10.318989 888 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.263370 823 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.442118 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.443809 953 consensus_queue.cc:1035] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:10.276856 921 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.449440 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.661121 961 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.661306 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.661945 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:10.661759 961 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.663461 32507 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.663955 961 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:10.569953 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.509629 30788 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:10.672622 30788 raft_consensus.cc:595] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:10.673157 30788 raft_consensus.cc:603] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:16:10.673782 30787 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:10.668912 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet 5ba1969a3911423e8e51fc4f2962515d
W20260806 20:16:10.669566 30757 leader_election.cc:334] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.922s (SENT)
I20260806 20:16:10.674986 30757 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:10.669727 30783 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:10.671401 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:10.677692 30705 raft_consensus.cc:2362] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8. Re-sending same reply.
I20260806 20:16:10.674546 30786 rpcz_store.cc:276] 0806 20:16:08.508453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.508560 (+ 107us) service_pool.cc:227] Handling call
0806 20:16:08.508877 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:09.678547 (+1169670us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:09.678578 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:09.691167 (+ 12589us) log.cc:842] Serialized 26 byte log entry
0806 20:16:10.052715 (+361548us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:10.052751 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:10.382374 (+329623us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54003b1fd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:10.382424 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:10.382473 (+ 49us) raft_consensus.cc:1671] finished
0806 20:16:10.382485 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:10.382704 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":517536,"raft.run_cpu_time_us":47,"raft.run_wall_time_us":47,"spinlock_wait_cycles":774401280,"thread_start_us":812452,"threads_started":2,"wal-append.queue_time_us":401301}
I20260806 20:16:10.677657 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.679296 30866 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:10.685561 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.675675 30711 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:10.687168 30866 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:10.449306 944 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:10.508621 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.688972 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.667829 878 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.439750 877 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:10.691507 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:10.692262 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 249918us
I20260806 20:16:10.456203 31421 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:10.693204 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:10.691615 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.712704 30866 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:10.713100 30870 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:10.692412 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:10.713970 30868 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:10.661509 957 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:10.670572 926 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:10.700079 30789 raft_consensus.cc:1261] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:16:10.722415 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.520442 30676 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:10.669220 879 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:10.705204 30861 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:16:10.706384 30835 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.440069 810 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:10.735098 810 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:10.735620 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.663976 32507 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.737643 690 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:16:10.738350 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 68825.3us
I20260806 20:16:10.735275 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:10.739007 30757 leader_election.cc:334] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.945s (SENT)
I20260806 20:16:10.692495 809 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:10.538705 32733 consensus_queue.cc:1035] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.187s
I20260806 20:16:10.741016 30756 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.742864 30865 raft_consensus.cc:2379] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:10.510200 872 consensus_queue.cc:1035] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.204s
I20260806 20:16:10.719641 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:10.444710 865 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.678016 973 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:10.675565 935 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:10.728029 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 207654us
I20260806 20:16:10.730247 31389 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:10.508955 608 raft_consensus.cc:686] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:10.734630 31414 raft_consensus.cc:2379] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 9: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:10.699640 30784 raft_consensus.cc:2396] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 20:16:10.745887 30870 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:10.445766 871 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.748292 30787 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:10.751206 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.753811 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.748034 809 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:10.740080 957 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.786372 608 consensus_queue.cc:237] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.787256 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.788372 957 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.717008 878 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.757180 618 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.738593 32507 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.757622 619 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:10.741788 879 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:10.758082 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.760506 30868 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9.
I20260806 20:16:10.795493 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.796118 30868 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:10.761569 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1433) took 2254 ms. Trace:
W20260806 20:16:10.797197 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 67033.7us
I20260806 20:16:10.761505 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:10.798496 31423 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:10.766361 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.799101 30703 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:10.799321 30832 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:10.445068 914 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.749954 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:10.771842 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:10.773273 857 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:16:10.774915 30757 leader_election.cc:334] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.147s (SENT)
I20260806 20:16:10.796811 30785 rpcz_store.cc:276] 0806 20:16:08.507333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:08.507484 (+ 151us) service_pool.cc:227] Handling call
0806 20:16:08.889615 (+382131us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:09.887753 (+998138us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:09.887783 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:09.888230 (+ 447us) log.cc:842] Serialized 26 byte log entry
0806 20:16:10.080944 (+192714us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:10.080975 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:10.081035 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:10.152515 (+ 71480us) raft_consensus.cc:1671] finished
0806 20:16:10.152535 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:10.761248 (+608713us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b54001419d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:10.761480 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":473660,"raft.run_cpu_time_us":42,"raft.run_wall_time_us":42,"spinlock_wait_cycles":764061568,"thread_start_us":1189994,"threads_started":2,"wal-append.queue_time_us":262543}
I20260806 20:16:10.810237 30757 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:10.777119 30861 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 8: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:10.749181 872 consensus_queue.cc:1225] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer 58d59beb597f4ed99d84d102ddb41ff5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:16:10.774470 812 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:10.781002 667 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:10.782066 30756 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.826301 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a439c5debb41440c913a50da91e6467f"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:10.782406 888 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.782851 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.783425 823 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.496189 938 consensus_queue.cc:1035] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.247s
I20260806 20:16:10.568756 954 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.550055 862 consensus_queue.cc:237] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.827009 30787 raft_consensus.cc:595] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:16:10.829560 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:10.829538 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.842195 30787 raft_consensus.cc:603] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:16:10.829646 30784 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:10.842628 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.740021 926 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.833026 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:10.850476 30707 pending_rounds.cc:76] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 1
I20260806 20:16:10.850584 926 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.764719 865 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.838503 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.852341 30703 raft_consensus.cc:1261] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:16:10.853188 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.854674 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.789706 973 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.855420 30868 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 8: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:10.856256 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.856760 973 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:10.839203 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 57222.1us
I20260806 20:16:10.862179 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:10.863087 969 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:10.864122 30756 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.840118 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.865478 30861 raft_consensus.cc:2379] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:10.866389 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.812040 885 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:10.870548 885 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.813890 32507 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:10.871785 32507 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:10.872226 32507 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:10.873970 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 99076.1us
I20260806 20:16:10.846132 972 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:10.846712 878 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:10.847029 921 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.834224 667 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.879769 30756 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:10.869860 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.874140 30707 pending_rounds.cc:97] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:16:10.815140 920 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:10.810003 857 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:10.882300 30861 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:10.887737 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.866716 888 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:16:10.874646 30757 leader_election.cc:334] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.245s (SENT)
I20260806 20:16:10.831184 812 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:10.921504 32507 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.927054 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.869331 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.883334 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.883656 30707 raft_consensus.cc:2866] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:10.882637 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.890121 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.928465 30870 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:10.929760 32507 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.894126 831 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:10.895815 667 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:10.896198 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:10.940729 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 66112.7us
I20260806 20:16:10.874703 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.941955 30868 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:10.885972 31423 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:10.842124 869 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:16:10.887095 30756 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.943764 30673 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:16:10.944051 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 57026.7us
I20260806 20:16:10.940745 30865 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:10.949556 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.951485 30834 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:10.952306 32507 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:10.952772 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:10.953961 30834 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:10.960371 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:10.961344 30868 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 8: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:10.948779 30756 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:10.926507 857 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:10.929345 812 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:10.910493 944 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.912535 914 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.888417 920 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:10.880635 959 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:10.914889 954 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:10.981307 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 274972us
I20260806 20:16:10.873376 810 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:10.921525 888 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:10.898283 921 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:10.973830 869 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:10.931660 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:10.932773 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:10.937855 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:10.938874 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:11.048851 888 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:11.053957 878 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:10.904076 871 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.969892 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:10.976310 32507 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:10.980466 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.066247 30865 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 8.
I20260806 20:16:11.067158 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:10.921689 809 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:10.919337 936 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:10.971243 667 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:10.949805 943 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:10.940140 31417 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 1.
I20260806 20:16:10.969358 898 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:11.027974 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:10.963562 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.047224 865 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:11.018942 920 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:11.018953 959 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:11.051232 30866 raft_consensus.cc:2354] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5.
I20260806 20:16:10.902045 30711 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
W20260806 20:16:11.020251 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 45972.5us
W20260806 20:16:11.059695 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 266840us
I20260806 20:16:11.058796 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.072269 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.058416 30861 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:11.061658 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:11.073104 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.162200 30713 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:11.162928 30784 raft_consensus.cc:2396] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 20:16:11.075585 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:11.165324 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.103505 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:11.102593 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 158918us
W20260806 20:16:11.095243 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 141343us
I20260806 20:16:11.067938 30865 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:11.099905 31422 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:16:11.171245 30834 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.121413 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.129279 30835 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:11.130455 30869 raft_consensus.cc:2354] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 7. Current term is 8.
I20260806 20:16:11.028496 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:11.055411 885 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:11.056196 869 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:11.050954 935 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:11.243116 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 64362.8us
I20260806 20:16:11.151111 943 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:11.252434 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:11.176318 959 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:11.119056 690 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:11.178010 31421 raft_consensus.cc:1226] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:11.181697 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:11.267930 30713 raft_consensus.cc:2396] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 20:16:11.182015 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:11.183118 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.184429 32507 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:11.123579 810 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:11.184376 30834 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.100741 619 consensus_queue.cc:1035] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
W20260806 20:16:11.183873 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 54708.9us
I20260806 20:16:11.271116 30870 raft_consensus.cc:2354] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5.
I20260806 20:16:11.175947 942 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:11.191066 972 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:11.159207 667 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.202212 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:16:11.277275 32507 raft_consensus.cc:686] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:16:11.277767 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 93472.9us
I20260806 20:16:11.202893 31414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49046 (request call id 598) took 1122 ms. Trace:
I20260806 20:16:11.158071 30676 leader_election.cc:398] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8.
I20260806 20:16:11.219274 809 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:11.220161 30790 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:11.262378 30711 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 20:16:11.271297 30835 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:11.192157 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.194003 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.241307 869 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:11.156950 938 consensus_queue.cc:1046] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260806 20:16:11.281539 31414 rpcz_store.cc:276] 0806 20:16:10.080393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.091851 (+ 11458us) service_pool.cc:227] Handling call
0806 20:16:11.202657 (+1110806us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54004ed4d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:11.202825 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1232804096}
I20260806 20:16:11.281427 30834 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.286270 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.290860 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:11.282904 30868 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:11.257558 943 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:11.283493 30676 leader_election.cc:401] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260806 20:16:11.316887 30703 raft_consensus.cc:2354] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5.
I20260806 20:16:11.317317 30787 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:11.320393 30834 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:11.322531 30834 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.216935 857 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.326413 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.250542 621 consensus_queue.cc:1035] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260806 20:16:11.329344 30834 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.285115 690 raft_consensus.cc:686] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:11.337280 31388 leader_election.cc:398] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5.
I20260806 20:16:11.337682 31388 leader_election.cc:401] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 20:16:11.325426 32507 consensus_queue.cc:237] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.337347 30835 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:11.341105 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 183161us
I20260806 20:16:11.342378 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:11.343922 957 consensus_queue.cc:1035] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:11.345527 810 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:11.345999 667 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:11.284137 942 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.347883 942 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:11.348739 898 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 20:16:11.298866 877 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:11.353363 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:11.355185 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 111071us
I20260806 20:16:11.356626 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.307785 885 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:11.361394 30703 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:16:11.392239 30789 raft_consensus.cc:2396] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260806 20:16:11.393396 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 51103.3us
I20260806 20:16:11.368155 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.383098 30832 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:11.390112 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.394940 30868 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 8.
I20260806 20:16:11.397362 30865 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:11.399629 30785 raft_consensus.cc:2379] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:16:11.399907 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 70655us
I20260806 20:16:11.401104 30834 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.402436 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:11.403982 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 66730us
I20260806 20:16:11.408177 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.409061 30868 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:11.411226 30792 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:11.414250 31389 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.415167 30835 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.410130 30861 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:11.349948 973 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:11.358719 914 raft_consensus.cc:3032] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:11.358982 865 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8.
I20260806 20:16:11.396595 838 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:11.389917 885 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:11.434772 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.440702 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.412848 755 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:11.402814 986 raft_consensus.cc:2741] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260806 20:16:11.448227 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 65216.8us
I20260806 20:16:11.409663 944 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:11.444716 987 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:11.484638 973 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:11.426875 964 consensus_queue.cc:1046] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
W20260806 20:16:11.477077 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 308155us
I20260806 20:16:11.488440 914 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:11.427496 619 consensus_queue.cc:1046] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260806 20:16:11.452102 877 raft_consensus.cc:2781] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:16:11.509704 885 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:11.512583 972 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:11.483037 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.553332 30791 raft_consensus.cc:2396] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260806 20:16:11.483687 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:11.451944 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:16:11.508698 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 68086.3us
I20260806 20:16:11.423125 690 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:11.446436 957 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:11.519423 998 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:16:11.520766 924 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:11.462255 753 consensus_queue.cc:1035] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260806 20:16:11.477712 30787 raft_consensus.cc:1261] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:16:11.524693 30757 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:11.488765 935 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:11.506726 862 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 20:16:11.574115 888 consensus_meta.cc:287] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.512s user 0.010s sys 0.000s
I20260806 20:16:11.574800 888 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:11.578541 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 53956.3us
I20260806 20:16:11.563055 31417 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:11.580751 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:11.447489 988 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:11.513713 755 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:11.482998 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.555514 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 20:16:11.632872 30788 raft_consensus.cc:2396] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 10 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 20:16:11.516901 898 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:11.642093 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:11.556686 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 142541us
I20260806 20:16:11.557579 30834 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:11.646003 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 88543.6us
I20260806 20:16:11.646759 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:11.648784 30834 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.524899 973 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:11.560817 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:11.654870 30791 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:11.647574 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:11.655061 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.655449 31421 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 5
W20260806 20:16:11.561641 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 157972us
I20260806 20:16:11.532177 877 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:16:11.563868 994 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.104s user 0.000s sys 0.001s
W20260806 20:16:11.569612 989 log.cc:924] Time spent T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.263s user 0.001s sys 0.001s
I20260806 20:16:11.530962 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:11.681910 31421 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:11.650647 30834 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.683889 810 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:11.684581 810 consensus_queue.cc:1035] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:11.685178 810 consensus_queue.cc:1225] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer 42cabca48db7403b8523a3382a046043 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 20:16:11.572041 924 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:11.682075 30785 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:11.688452 924 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:11.668196 898 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.693912 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:11.694711 30787 raft_consensus.cc:3037] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:11.585809 999 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:11.699503 999 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:11.699932 999 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:11.702847 812 consensus_meta.cc:287] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.680s user 0.007s sys 0.002s
I20260806 20:16:11.703321 812 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.580106 914 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.713112 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:11.714783 30785 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:11.719700 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:11.670969 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.673641 809 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:11.674365 857 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:11.674618 1001 log.cc:924] Time spent T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260806 20:16:11.639649 755 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:11.676048 993 log.cc:924] Time spent T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.165s user 0.000s sys 0.000s
I20260806 20:16:11.581195 888 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:11.676682 997 log.cc:924] Time spent T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.227s user 0.000s sys 0.001s
I20260806 20:16:11.654369 972 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:11.671929 877 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:16:11.668222 987 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:11.680426 873 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:11.570374 879 consensus_meta.cc:287] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.744s user 0.011s sys 0.001s
I20260806 20:16:11.724056 30832 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:16:11.784010 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 60067.7us
I20260806 20:16:11.785071 809 raft_consensus.cc:2781] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:11.728782 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:11.711269 30791 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:11.732544 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:11.675335 992 log.cc:924] Time spent T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.159s user 0.000s sys 0.000s
I20260806 20:16:11.753295 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.792410 30785 raft_consensus.cc:2396] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260806 20:16:11.769752 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 344901us
I20260806 20:16:11.771641 30787 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:11.793942 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:16:11.772611 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 189337us
I20260806 20:16:11.705317 812 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:11.706453 914 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:11.773768 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:11.779527 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.782960 943 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.783138 879 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.789746 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:11.795241 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.798496 30712 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 8.
W20260806 20:16:11.801718 30783 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.019s sys 0.015s
W20260806 20:16:11.802129 30783 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s user 0.019s sys 0.015s
W20260806 20:16:11.805795 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 77097.5us
I20260806 20:16:11.806699 30835 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:11.807653 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 124208us
W20260806 20:16:11.808406 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 99419.1us
I20260806 20:16:11.810501 30791 raft_consensus.cc:1261] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 20:16:11.811674 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:11.816581 30864 raft_consensus.cc:2379] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:11.817325 30789 raft_consensus.cc:2354] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7.
I20260806 20:16:11.818032 30834 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:11.812426 987 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:11.826077 31388 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:11.833374 987 raft_consensus.cc:686] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:11.839139 987 consensus_queue.cc:237] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:11.839195 31417 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:11.855639 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 191) took 1180 ms. Trace:
I20260806 20:16:11.856114 30861 rpcz_store.cc:276] 0806 20:16:10.674880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.491437 (+816557us) service_pool.cc:227] Handling call
0806 20:16:11.491762 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:11.509183 (+ 17421us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:11.515726 (+ 6543us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:11.562537 (+ 46811us) log.cc:842] Serialized 26 byte log entry
0806 20:16:11.773611 (+211074us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:11.799564 (+ 25953us) log.cc:842] Serialized 14 byte log entry
0806 20:16:11.801062 (+ 1498us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:11.801150 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:11.853385 (+ 52235us) spinlock_profiling.cc:243] Waited 8.83 ms on lock 0x7b18004476a0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e753638 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d 00007f71579ab204 00007f71579ab039 00007f714e735dc6
0806 20:16:11.855344 (+ 1959us) raft_consensus.cc:1671] finished
0806 20:16:11.855444 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:11.855575 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1236,"prepare.queue_time_us":28394,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":2737,"raft.queue_time_us":13108,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":11165,"spinlock_wait_cycles":25449088,"thread_start_us":74961,"threads_started":1,"wal-append.queue_time_us":153598}
I20260806 20:16:11.858309 1021 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:11.858707 30861 raft_consensus.cc:1261] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:16:11.858876 1021 raft_consensus.cc:686] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:11.859891 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1503) took 1018 ms. Trace:
I20260806 20:16:11.859799 1021 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:11.860334 30861 rpcz_store.cc:276] 0806 20:16:10.841128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.858069 (+1016941us) service_pool.cc:227] Handling call
0806 20:16:11.858382 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:11.859640 (+ 1258us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:11.859837 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:11.862861 943 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:11.915495 30861 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260806 20:16:11.916535 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1510) took 1018 ms. Trace:
I20260806 20:16:11.916980 30861 rpcz_store.cc:276] 0806 20:16:10.898156 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.914776 (+1016620us) service_pool.cc:227] Handling call
0806 20:16:11.915216 (+ 440us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:11.916218 (+ 1002us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:11.916474 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:12.023404 872 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:12.024205 872 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.027416 872 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:11.814353 690 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260806 20:16:11.833549 924 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:11.832700 810 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:16:11.957921 997 log.cc:924] Time spent T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260806 20:16:11.835815 898 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:12.067924 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 193) took 1392 ms. Trace:
I20260806 20:16:12.070098 30866 rpcz_store.cc:276] 0806 20:16:10.675379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.713500 (+ 38121us) service_pool.cc:227] Handling call
0806 20:16:12.067692 (+1354192us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b54001e4954. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:12.067858 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":702873344}
W20260806 20:16:12.042081 992 log.cc:924] Time spent T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.205s user 0.000s sys 0.000s
I20260806 20:16:12.074550 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1635) took 1372 ms. Trace:
I20260806 20:16:12.074994 30705 rpcz_store.cc:276] 0806 20:16:10.702290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.702457 (+ 167us) service_pool.cc:227] Handling call
0806 20:16:10.702853 (+ 396us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:10.706295 (+ 3442us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:10.706343 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:10.786238 (+ 79895us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b48001c0158. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71565167f1 00007f715650d8e7 00007f715650d77a 00007f715660195f 00007f7156601627 00007f7156615b2c 00007f7156615ad8 00007f7156615a58 00007f7156615a13 00007f7156614cbc 00007f7155596be6
0806 20:16:10.853893 (+ 67655us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b2c00055798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:16:10.854387 (+ 494us) log.cc:842] Serialized 26 byte log entry
0806 20:16:10.979620 (+125233us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:11.093692 (+114072us) log.cc:842] Serialized 14 byte log entry
0806 20:16:11.116684 (+ 22992us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:11.116766 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:12.074322 (+957556us) raft_consensus.cc:1671] finished
0806 20:16:12.074342 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:12.074489 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32774,"prepare.queue_time_us":113840,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":26732,"raft.run_cpu_time_us":1934,"raft.run_wall_time_us":84724,"spinlock_wait_cycles":21988224,"thread_start_us":124834,"threads_started":1,"wal-append.queue_time_us":598896}
I20260806 20:16:11.849143 999 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.050345 924 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:16:12.092332 869 consensus_meta.cc:287] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.767s user 0.010s sys 0.000s
W20260806 20:16:12.094559 838 consensus_meta.cc:287] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.583s user 0.009s sys 0.000s
W20260806 20:16:12.108359 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 286292us
I20260806 20:16:12.093205 924 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:12.124049 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:12.124955 30787 raft_consensus.cc:2379] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:16:12.129232 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1374) took 1469 ms (client timeout 1922 ms). Trace:
I20260806 20:16:12.129362 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 199) took 1433 ms. Trace:
W20260806 20:16:12.129940 30869 rpcz_store.cc:269] 0806 20:16:10.660084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.661687 (+ 1603us) service_pool.cc:227] Handling call
0806 20:16:12.129027 (+1467340us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b540014c3d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:12.129169 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1075737344}
I20260806 20:16:12.130308 30870 rpcz_store.cc:276] 0806 20:16:10.696063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.981081 (+285018us) service_pool.cc:227] Handling call
0806 20:16:12.129192 (+1148111us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540019da54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:12.129308 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":251100544}
I20260806 20:16:12.110961 838 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937) with higher term. Message: Invalid argument: T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5.
I20260806 20:16:12.138166 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1375) took 1477 ms. Trace:
W20260806 20:16:12.115983 995 log.cc:924] Time spent T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.780s user 0.015s sys 0.000s
I20260806 20:16:12.153757 898 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:12.154428 898 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.156663 898 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:12.158002 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1641) took 1450 ms. Trace:
W20260806 20:16:12.127616 885 consensus_meta.cc:287] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.567s user 0.010s sys 0.000s
I20260806 20:16:12.165016 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:12.167961 30783 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:16:12.167407 30707 rpcz_store.cc:276] 0806 20:16:10.707474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.849825 (+142351us) service_pool.cc:227] Handling call
0806 20:16:10.850231 (+ 406us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:10.956508 (+106277us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:10.956542 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:10.957175 (+ 633us) log.cc:842] Serialized 52 byte log entry
0806 20:16:11.284507 (+327332us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:11.335819 (+ 51312us) log.cc:842] Serialized 14 byte log entry
0806 20:16:11.349991 (+ 14172us) log.cc:842] Serialized 14 byte log entry
0806 20:16:11.391178 (+ 41187us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:11.391269 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:11.854185 (+462916us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b180042ef80. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e753764 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d 00007f71579ab204 00007f71579ab039 00007f714e735dc6
0806 20:16:12.157764 (+303579us) raft_consensus.cc:1671] finished
0806 20:16:12.157798 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:12.157941 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64152,"prepare.queue_time_us":57956,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":78652,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":56093,"spinlock_wait_cycles":36033792,"thread_start_us":264790,"threads_started":1,"wal-append.queue_time_us":445289}
W20260806 20:16:12.166286 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 348315us
I20260806 20:16:12.147315 30863 rpcz_store.cc:276] 0806 20:16:10.660409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.662598 (+ 2189us) service_pool.cc:227] Handling call
0806 20:16:10.662968 (+ 370us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:10.663422 (+ 454us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:10.663452 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:10.664015 (+ 563us) log.cc:842] Serialized 52 byte log entry
0806 20:16:11.160462 (+496447us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:11.454731 (+294269us) log.cc:842] Serialized 14 byte log entry
0806 20:16:11.455162 (+ 431us) log.cc:842] Serialized 14 byte log entry
0806 20:16:11.455385 (+ 223us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:11.455448 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:12.137897 (+682449us) raft_consensus.cc:1671] finished
0806 20:16:12.137923 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:12.138100 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":105649,"prepare.queue_time_us":420787,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":14861,"raft.run_cpu_time_us":406,"raft.run_wall_time_us":24909,"thread_start_us":673169,"threads_started":2,"wal-append.queue_time_us":789253}
I20260806 20:16:12.107080 869 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.178572 30863 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:12.167891 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:12.177762 30756 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:12.176487 30835 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:12.148682 879 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:12.186884 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.187703 30707 raft_consensus.cc:2379] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:12.191007 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.164654 809 raft_consensus.cc:686] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:12.193742 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.195035 30788 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
W20260806 20:16:12.188189 959 consensus_meta.cc:287] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.901s user 0.008s sys 0.004s
I20260806 20:16:12.201581 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.203896 30703 pending_rounds.cc:76] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 3
I20260806 20:16:12.205787 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.213948 30712 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 8
I20260806 20:16:12.214085 30703 pending_rounds.cc:97] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260806 20:16:12.216205 30703 raft_consensus.cc:2866] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:12.187536 857 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 20:16:12.212016 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 261157us
I20260806 20:16:12.218698 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.220526 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.211361 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.187559 810 raft_consensus.cc:686] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:12.223184 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.229664 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.236219 30712 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:12.174827 885 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.239992 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.244695 31389 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:12.245846 30706 raft_consensus.cc:1261] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 20:16:12.247488 30787 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:12.250607 1029 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:12.251075 1029 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:12.251416 1029 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:12.264599 31389 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.227763 857 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:12.267571 30868 raft_consensus.cc:1226] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:12.271627 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:12.274108 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 83553.7us
W20260806 20:16:12.289762 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 104099us
I20260806 20:16:12.292430 30757 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:12.219511 959 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.307166 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1384) took 1633 ms. Trace:
I20260806 20:16:12.212512 1023 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260806 20:16:12.274333 857 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:12.307651 30871 rpcz_store.cc:276] 0806 20:16:10.673519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.673625 (+ 106us) service_pool.cc:227] Handling call
0806 20:16:10.699098 (+ 25473us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:10.700751 (+ 1653us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:10.700780 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:10.724533 (+ 23753us) log.cc:842] Serialized 26 byte log entry
0806 20:16:10.873705 (+149172us) spinlock_profiling.cc:243] Waited 5.34 ms on lock 0x7b1800259780. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e735345 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb 00007f71512f6063 00007f71512f6411 00007f71516435cf
0806 20:16:11.136861 (+263156us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:16:11.464100 (+327239us) log.cc:842] Serialized 14 byte log entry
0806 20:16:11.464395 (+ 295us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:11.464477 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:11.834621 (+370144us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:12.306915 (+472294us) raft_consensus.cc:1671] finished
0806 20:16:12.306944 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:12.307101 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11169,"prepare.queue_time_us":372462,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":14851,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":11110,"spinlock_wait_cycles":15687680,"thread_start_us":552139,"threads_started":2,"wal-append.queue_time_us":551196}
I20260806 20:16:12.269352 1007 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:16:12.310094 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1518) took 1075 ms. Trace:
W20260806 20:16:12.311729 973 consensus_meta.cc:287] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.638s user 0.004s sys 0.007s
I20260806 20:16:12.314524 30866 rpcz_store.cc:276] 0806 20:16:11.234217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.080539 (+846322us) service_pool.cc:227] Handling call
0806 20:16:12.098346 (+ 17807us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:12.098576 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:16:12.098899 (+ 323us) log.cc:842] Serialized 14 byte log entry
0806 20:16:12.309749 (+210850us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:12.309804 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:12.309852 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:12.309917 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:12.310035 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2093,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":58076,"thread_start_us":210443,"threads_started":2}
I20260806 20:16:12.315637 973 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.273036 999 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:12.321022 30790 raft_consensus.cc:1226] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:12.333267 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.336784 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.337551 30866 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:12.269642 1002 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.062s
I20260806 20:16:12.314230 862 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:12.319437 1034 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:16:12.378604 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 540928us
I20260806 20:16:12.382251 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.332747 1007 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:12.325664 1027 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:12.383275 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 111710us
I20260806 20:16:12.338811 838 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:12.394519 30787 raft_consensus.cc:1261] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:12.283298 809 consensus_queue.cc:237] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.397991 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.396739 31389 leader_election.cc:398] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5.
I20260806 20:16:12.398787 31389 leader_election.cc:401] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:16:12.403537 30866 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:12.404613 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:12.397161 30713 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.010s sys 0.006s
W20260806 20:16:12.405831 30713 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.010s sys 0.006s
I20260806 20:16:12.390756 1034 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.415237 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1497) took 1644 ms. Trace:
I20260806 20:16:12.415629 30868 rpcz_store.cc:276] 0806 20:16:10.770492 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.557915 (+787423us) service_pool.cc:227] Handling call
0806 20:16:11.558236 (+ 321us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:12.275265 (+717029us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:12.414998 (+139733us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b540014bc54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:12.415168 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1630890752}
I20260806 20:16:12.416808 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.418383 30868 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:12.419246 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.419991 30868 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 8: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:12.389707 869 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:12.425325 920 consensus_meta.cc:287] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.186s user 0.013s sys 0.000s
I20260806 20:16:12.380900 1033 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:12.425869 920 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.427255 31419 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:12.428699 920 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:12.429813 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:12.430632 31416 raft_consensus.cc:2379] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:12.393781 1027 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.435423 1027 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:12.436594 1032 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:12.440970 30834 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.442601 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.443248 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.445966 1007 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5.
I20260806 20:16:12.446846 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:12.319306 810 consensus_queue.cc:237] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.451305 1030 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:12.452015 1030 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:12.454329 1031 log.cc:924] Time spent T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260806 20:16:12.455228 1030 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:12.456279 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.457136 973 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:12.457659 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.458510 30866 raft_consensus.cc:2379] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:12.458365 959 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:12.414460 755 consensus_meta.cc:287] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.638s user 0.010s sys 0.000s
I20260806 20:16:12.459898 885 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:12.460549 885 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:12.461297 31421 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:16:12.393198 838 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:12.465736 838 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:12.467973 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.468719 30712 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:12.470335 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:16:12.471269 31420 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 402) took 1537 ms (client timeout 1733 ms). Trace:
W20260806 20:16:12.471662 31420 rpcz_store.cc:269] 0806 20:16:10.933522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.938417 (+ 4895us) service_pool.cc:227] Handling call
0806 20:16:12.470136 (+1531719us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b5800005198. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:16:12.471203 (+ 1067us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91876608}
I20260806 20:16:12.475489 30834 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:12.476437 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.478662 30866 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:12.422788 1023 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:12.480860 1023 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:12.481230 1023 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:12.483402 936 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:16:12.485965 30834 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.475175 924 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.487915 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:16:12.488528 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 205242us
I20260806 20:16:12.491437 30756 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:12.425568 1026 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
W20260806 20:16:12.492882 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 200551us
I20260806 20:16:12.487676 30834 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.495617 30712 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:16:12.497884 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.498073 30706 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260806 20:16:12.498811 30832 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.009s sys 0.029s
W20260806 20:16:12.499866 30832 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.009s sys 0.029s
I20260806 20:16:12.500865 30757 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:12.501425 30834 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.503386 30757 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:12.504978 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.505776 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.507692 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:16:12.491595 877 consensus_meta.cc:287] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.697s user 0.004s sys 0.007s
I20260806 20:16:12.459308 755 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.531821 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.534471 30707 raft_consensus.cc:3037] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 7
I20260806 20:16:12.528659 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.530066 30834 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.540023 30710 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260806 20:16:12.536695 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.497398 973 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:16:12.554445 914 consensus_meta.cc:287] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.754s user 0.010s sys 0.001s
I20260806 20:16:12.555294 30711 raft_consensus.cc:2396] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 20:16:12.559787 1047 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:12.562076 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1516) took 1328 ms. Trace:
I20260806 20:16:12.562503 30865 rpcz_store.cc:276] 0806 20:16:11.233462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.285624 (+ 52162us) service_pool.cc:227] Handling call
0806 20:16:12.561902 (+1276278us) spinlock_profiling.cc:243] Waited 89.1 ms on lock 0x7b540019d054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:12.562016 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":204916992}
I20260806 20:16:12.563180 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1520) took 1328 ms. Trace:
I20260806 20:16:12.501879 1034 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:12.540563 877 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.495076 898 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:12.563552 30869 rpcz_store.cc:276] 0806 20:16:11.234771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.147784 (+913013us) service_pool.cc:227] Handling call
0806 20:16:12.170959 (+ 23175us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:12.171280 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:12.171307 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:12.171933 (+ 626us) log.cc:842] Serialized 26 byte log entry
0806 20:16:12.349758 (+177825us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:12.349805 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:12.349877 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:12.562961 (+213084us) raft_consensus.cc:1671] finished
0806 20:16:12.562982 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:12.563122 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17507,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":10301,"thread_start_us":169866,"threads_started":1,"wal-append.queue_time_us":317186}
I20260806 20:16:12.508750 869 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:16:12.548743 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:12.598982 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 95676.7us
I20260806 20:16:12.550076 30834 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.426016 1033 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:12.566102 991 log.cc:924] Time spent T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 1.042s user 0.002s sys 0.019s
I20260806 20:16:12.552474 755 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:12.526643 1012 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.036s sys 0.060s
I20260806 20:16:12.553787 1042 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:12.519974 961 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 20:16:12.545815 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 151211us
I20260806 20:16:12.563565 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.563813 1040 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 20:16:12.565860 1045 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:12.548623 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.599874 30757 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:12.551275 31418 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:12.603396 30866 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:12.604936 924 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:12.605221 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.611200 30707 raft_consensus.cc:2454] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:12.567349 608 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
W20260806 20:16:12.619558 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 88395.5us
I20260806 20:16:12.619796 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.566709 877 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:12.624436 30865 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:12.563454 30706 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:12.637715 30713 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:12.627902 30757 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:12.628451 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:12.626890 30832 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:12.628377 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 59105.6us
I20260806 20:16:12.638916 30757 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:16:12.629004 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 78999.8us
I20260806 20:16:12.641412 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:12.630333 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.631623 961 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:12.563722 1021 consensus_queue.cc:1035] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260806 20:16:12.632431 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:16:12.619313 1012 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.756s user 0.037s sys 0.060s
I20260806 20:16:12.633244 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.645671 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.646066 30868 raft_consensus.cc:2362] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9. Re-sending same reply.
I20260806 20:16:12.637144 30869 raft_consensus.cc:2379] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:12.646662 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.626163 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:12.647017 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.643173 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:12.634334 1029 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.607309 1023 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.649833 30865 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:12.658392 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.657150 30713 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 9.
I20260806 20:16:12.628868 869 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:12.663044 869 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:12.663451 869 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:12.663784 869 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:12.666062 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.666971 30869 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:16:12.618986 857 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.651041 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.652007 30871 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:12.652782 30832 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.655531 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.670184 30712 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:12.660496 30865 raft_consensus.cc:2362] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9. Re-sending same reply.
I20260806 20:16:12.661031 1048 consensus_queue.cc:1046] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:16:12.680660 1042 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 20:16:12.682621 31388 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.683109 1023 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:12.683470 924 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:16:12.684047 924 raft_consensus.cc:686] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:12.652169 1002 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:12.684932 924 consensus_queue.cc:237] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.686841 30792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1851) took 1613 ms. Trace:
I20260806 20:16:12.687704 857 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:12.637667 1045 raft_consensus.cc:686] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:12.690407 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.690460 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.687207 30792 rpcz_store.cc:276] 0806 20:16:11.073752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.074494 (+ 742us) service_pool.cc:227] Handling call
0806 20:16:11.074798 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:11.424032 (+349234us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:12.686616 (+1262584us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b5400260554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:12.686781 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":773263616}
I20260806 20:16:12.672458 30713 raft_consensus.cc:1261] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 20:16:12.694600 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.694985 30866 raft_consensus.cc:2354] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7.
I20260806 20:16:12.695472 30710 raft_consensus.cc:2379] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:12.692301 30865 raft_consensus.cc:2362] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5. Re-sending same reply.
I20260806 20:16:12.698931 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.644512 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:12.648192 810 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:12.734021 31416 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:12.736878 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1512) took 1786 ms. Trace:
I20260806 20:16:12.737653 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.747045 30861 rpcz_store.cc:276] 0806 20:16:10.950506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.029192 (+1078686us) service_pool.cc:227] Handling call
0806 20:16:12.029706 (+ 514us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:12.186098 (+156392us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:12.186129 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:12.220762 (+ 34633us) log.cc:842] Serialized 52 byte log entry
0806 20:16:12.518367 (+297605us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:12.557863 (+ 39496us) log.cc:842] Serialized 14 byte log entry
0806 20:16:12.590723 (+ 32860us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:12.590813 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:12.736634 (+145821us) raft_consensus.cc:1671] finished
0806 20:16:12.736653 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:12.736816 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50598,"prepare.queue_time_us":39502,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":156674,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":9958,"thread_start_us":395995,"threads_started":2,"wal-append.queue_time_us":381192}
I20260806 20:16:12.750922 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.751705 30861 raft_consensus.cc:2379] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:12.753876 30832 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:12.779701 969 raft_consensus.cc:3032] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:12.780241 969 raft_consensus.cc:729] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:12.788803 30865 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:12.837298 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.839905 30868 raft_consensus.cc:2362] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7. Re-sending same reply.
W20260806 20:16:12.838225 30757 leader_election.cc:334] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.770s (SENT)
I20260806 20:16:12.842541 30757 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:12.843953 30757 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:12.846661 31422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 397) took 1978 ms. Trace:
W20260806 20:16:12.850977 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 97200us
I20260806 20:16:12.789007 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.847076 31422 rpcz_store.cc:276] 0806 20:16:10.868539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.868692 (+ 153us) service_pool.cc:227] Handling call
0806 20:16:12.846507 (+1977815us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54005d39d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:12.846608 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":392482944}
I20260806 20:16:12.857116 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.857767 30713 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:12.837157 1062 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:12.860918 1062 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.862095 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.866468 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.846195 921 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:12.867677 1005 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:12.824399 608 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:12.876099 30757 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:12.877934 30757 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:12.873764 30832 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:12.867056 30861 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:12.741488 30712 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:12.839226 1045 consensus_queue.cc:237] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.782089 961 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:16:12.891289 30676 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.092s user 0.005s sys 0.015s
W20260806 20:16:12.924454 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 50784.8us
I20260806 20:16:12.892238 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1854) took 1408 ms. Trace:
I20260806 20:16:12.925343 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:16:12.865327 1055 log.cc:924] Time spent T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
W20260806 20:16:12.929527 30676 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.132s user 0.020s sys 0.023s
I20260806 20:16:12.930325 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.931005 30790 rpcz_store.cc:276] 0806 20:16:11.484238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.802310 (+318072us) service_pool.cc:227] Handling call
0806 20:16:11.802616 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:12.325643 (+523027us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:12.891996 (+566353us) spinlock_profiling.cc:243] Waited 518 ms on lock 0x7b54003d59d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:12.892176 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1191735168}
W20260806 20:16:12.936117 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 292045us
I20260806 20:16:12.932188 30713 raft_consensus.cc:2454] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
W20260806 20:16:12.935410 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 1.13825e+06us
W20260806 20:16:12.935091 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 57227.3us
I20260806 20:16:12.939635 30757 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:12.941262 30757 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:12.937278 30756 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:12.943670 30757 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:12.874589 1033 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:12.945055 31422 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:16:12.947492 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.950208 30713 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 20:16:12.951655 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:12.946767 1002 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:12.953023 1023 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:12.954377 936 consensus_queue.cc:1046] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:12.955228 936 raft_consensus.cc:3032] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:12.955574 936 raft_consensus.cc:729] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:12.956302 936 consensus_queue.cc:260] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.957388 30871 raft_consensus.cc:1226] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:12.737304 1061 consensus_queue.cc:1046] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.247s
I20260806 20:16:12.934432 810 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:12.969336 506 consensus_queue.cc:1046] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.003s
I20260806 20:16:12.969376 1017 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260806 20:16:12.973157 506 raft_consensus.cc:3032] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:12.973603 506 raft_consensus.cc:729] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:16:12.974684 30864 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1379) took 2302 ms (client timeout 1741 ms). Trace:
W20260806 20:16:12.975090 30864 rpcz_store.cc:269] 0806 20:16:10.671877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.672002 (+ 125us) service_pool.cc:227] Handling call
0806 20:16:12.974445 (+2302443us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400375854. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:12.974585 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2570744576}
I20260806 20:16:12.976018 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:12.977310 1068 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:12.923899 608 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.978009 1068 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:12.978436 1068 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:12.979713 608 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:12.823019 969 consensus_queue.cc:260] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.976099 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:12.981585 30792 raft_consensus.cc:1261] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 20:16:12.982172 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.958281 935 consensus_queue.cc:1046] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:12.983947 935 raft_consensus.cc:3032] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:12.974438 506 consensus_queue.cc:260] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.984272 935 raft_consensus.cc:729] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:12.984923 506 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:12.985167 935 consensus_queue.cc:260] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.986485 935 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:12.986699 1002 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:12.987217 1002 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.987669 30870 raft_consensus.cc:1226] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 20:16:12.989306 1002 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:12.989682 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1521) took 1754 ms. Trace:
I20260806 20:16:12.887701 1005 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:12.933828 1066 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260806 20:16:12.964377 1023 raft_consensus.cc:686] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:12.944926 1062 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:12.948236 1063 raft_consensus.cc:2781] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:12.957384 936 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:12.966322 30871 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 20:16:12.989571 810 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:12.941366 869 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:12.990082 30870 rpcz_store.cc:276] 0806 20:16:11.235043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.147760 (+912717us) service_pool.cc:227] Handling call
0806 20:16:12.987434 (+839674us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:12.988094 (+ 660us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:12.989491 (+ 1397us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x7b5400375850. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:12.989623 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1930288512}
I20260806 20:16:12.888775 1034 consensus_queue.cc:1046] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260806 20:16:12.992095 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:12.993868 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:12.890223 838 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:12.967734 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:12.995523 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:12.997753 30792 raft_consensus.cc:1261] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:16:13.000180 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1524) took 1764 ms. Trace:
I20260806 20:16:12.946861 621 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:16:12.999624 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1376) took 2339 ms. Trace:
I20260806 20:16:13.005200 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:13.010054 30867 rpcz_store.cc:276] 0806 20:16:10.660695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:10.676769 (+ 16074us) service_pool.cc:227] Handling call
0806 20:16:10.699046 (+ 22277us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:10.788362 (+ 89316us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:10.788393 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:10.851931 (+ 63538us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b2c001497e8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:16:10.852306 (+ 375us) log.cc:842] Serialized 26 byte log entry
0806 20:16:11.145687 (+293381us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:11.436632 (+290945us) log.cc:842] Serialized 14 byte log entry
0806 20:16:11.445774 (+ 9142us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:12.565481 (+1119707us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:12.999311 (+433830us) spinlock_profiling.cc:243] Waited 89.1 ms on lock 0x7b5400376754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:12.999363 (+ 52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:12.999413 (+ 50us) raft_consensus.cc:1671] finished
0806 20:16:12.999426 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:12.999557 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37768,"prepare.queue_time_us":372133,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":9158,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":2057,"spinlock_wait_cycles":217226240,"thread_start_us":466930,"threads_started":2,"wal-append.queue_time_us":623843}
I20260806 20:16:13.011240 1023 consensus_queue.cc:237] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.012189 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:13.012385 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:13.012935 30867 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 8: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:13.013381 31419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 505) took 1849 ms. Trace:
I20260806 20:16:13.013753 31419 rpcz_store.cc:276] 0806 20:16:11.163921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.187429 (+ 23508us) service_pool.cc:227] Handling call
0806 20:16:11.187774 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:12.427598 (+1239824us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:13.013188 (+585590us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400602554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:13.013329 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2850075008}
I20260806 20:16:13.008548 30863 rpcz_store.cc:276] 0806 20:16:11.235921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.196760 (+960839us) service_pool.cc:227] Handling call
0806 20:16:12.216337 (+ 19577us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:12.216695 (+ 358us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:12.216724 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:12.229063 (+ 12339us) log.cc:842] Serialized 26 byte log entry
0806 20:16:12.531730 (+302667us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:12.531773 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:12.531846 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:12.999977 (+468131us) raft_consensus.cc:1671] finished
0806 20:16:12.999997 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:13.000124 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":56901,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":33673,"thread_start_us":254137,"threads_started":1,"wal-append.queue_time_us":473182}
I20260806 20:16:13.016923 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:13.017135 30757 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:13.017724 30863 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:13.020761 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:13.022668 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:13.024829 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:13.025552 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:13.024374 30871 raft_consensus.cc:2379] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 2: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:13.028373 30863 raft_consensus.cc:2354] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7.
I20260806 20:16:13.028525 30757 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:13.026821 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:13.029526 608 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:13.013708 690 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:13.030272 621 raft_consensus.cc:686] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 9 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:13.030048 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:13.031502 30871 raft_consensus.cc:2379] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 8: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:13.031232 621 consensus_queue.cc:237] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.032902 1028 consensus_queue.cc:1046] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:16:13.033653 1028 consensus_queue.cc:1225] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer 58f2f6256ee14cb38a1dbcbf5bb590a0 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 20:16:13.023878 914 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260806 20:16:13.036103 30599 catalog_manager.cc:5495] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 1 to 7. New cstate: current_term: 7 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:13.029937 608 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:13.002436 810 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:11.633036 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:16:13.019883 812 consensus_queue.cc:1035] T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
W20260806 20:16:13.040553 30788 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1461) took 1462 ms (client timeout 1844 ms). Trace:
W20260806 20:16:13.040980 30788 rpcz_store.cc:269] 0806 20:16:11.578527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.670665 (+ 92138us) service_pool.cc:227] Handling call
0806 20:16:13.040383 (+1369718us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b54001d24d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:13.040490 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1085801728}
I20260806 20:16:13.042702 869 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:13.045789 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:13.038208 608 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:13.046494 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:13.038367 30599 catalog_manager.cc:5495] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 5 to 6, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 6 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:13.048489 30863 raft_consensus.cc:1226] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 1. Current term is 2. Ops: []
I20260806 20:16:13.046480 30864 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:13.051393 31389 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.089210 30756 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:13.056077 30757 leader_election.cc:398] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7.
W20260806 20:16:13.102787 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 58292.8us
I20260806 20:16:13.052650 30867 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:13.102262 1061 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:13.020406 1063 raft_consensus.cc:686] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:16:13.107788 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 52337.7us
I20260806 20:16:13.123584 30756 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:13.125373 32507 raft_consensus.cc:3032] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:13.130743 32507 raft_consensus.cc:729] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:13.131598 32507 consensus_queue.cc:260] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.132800 32507 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:13.134557 30710 raft_consensus.cc:1261] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:16:13.122761 1061 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.137501 1061 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:13.138234 1063 consensus_queue.cc:237] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.125121 30757 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.103138 869 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:13.104633 999 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:16:13.143985 30864 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:16:13.148301 30711 raft_consensus.cc:3032] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:13.148679 30711 raft_consensus.cc:729] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:13.145150 30867 raft_consensus.cc:1261] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:16:13.102201 667 consensus_queue.cc:1046] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260806 20:16:13.150497 30757 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.152110 30711 consensus_queue.cc:260] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.160387 30711 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:13.180585 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 57067.9us
I20260806 20:16:13.146646 869 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
W20260806 20:16:13.209322 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 58902.5us
I20260806 20:16:13.387183 30757 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:13.388398 30757 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:13.388935 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 212748us
I20260806 20:16:13.390025 30757 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:13.210062 30834 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:13.212162 1072 log.cc:924] Time spent T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260806 20:16:13.393499 30711 raft_consensus.cc:1261] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260806 20:16:13.157629 809 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:13.151676 667 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:13.210393 30792 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:13.344671 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 327185us
W20260806 20:16:13.243999 888 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.896s user 0.003s sys 0.030s
I20260806 20:16:13.160073 1002 raft_consensus.cc:3032] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:13.127915 898 consensus_queue.cc:1035] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260806 20:16:13.160285 1061 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:13.170176 999 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:13.394474 30711 pending_rounds.cc:76] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 3
I20260806 20:16:13.395038 30757 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.399147 30676 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:13.399183 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 196994us
I20260806 20:16:13.402354 30711 pending_rounds.cc:97] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260806 20:16:13.400610 1042 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:13.400882 809 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.405273 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1525) took 2169 ms. Trace:
I20260806 20:16:13.409039 30711 raft_consensus.cc:2866] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:13.394593 869 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:13.382592 1023 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:13.411211 30834 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.399487 1028 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:16:13.399333 888 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.145s user 0.004s sys 0.051s
I20260806 20:16:13.416816 30861 rpcz_store.cc:276] 0806 20:16:11.236199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.947197 (+1710998us) service_pool.cc:227] Handling call
0806 20:16:12.947562 (+ 365us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:12.978280 (+ 30718us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:12.978314 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:12.978903 (+ 589us) log.cc:842] Serialized 52 byte log entry
0806 20:16:13.125411 (+146508us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:13.150988 (+ 25577us) log.cc:842] Serialized 14 byte log entry
0806 20:16:13.202460 (+ 51472us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:13.202546 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:13.405049 (+202503us) raft_consensus.cc:1671] finished
0806 20:16:13.405069 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:13.405214 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":243594,"prepare.queue_time_us":25884,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":79859,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":5103,"thread_start_us":146084,"threads_started":1,"wal-append.queue_time_us":423708}
I20260806 20:16:13.432982 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:13.399129 1005 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:13.425706 30834 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.420635 30757 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.427022 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:13.399399 1064 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:13.431471 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:13.408192 936 consensus_queue.cc:1035] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260806 20:16:13.429116 1028 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:13.400866 1002 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:13.436901 1061 consensus_queue.cc:1046] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:13.418524 865 raft_consensus.cc:2741] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260806 20:16:13.438422 996 log.cc:924] Time spent T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 1.927s user 0.004s sys 0.001s
I20260806 20:16:13.438678 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:13.430987 987 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:13.444260 30757 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.414306 1063 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:13.453878 30834 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.456748 1068 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.459991 871 raft_consensus.cc:2781] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 20:16:13.460235 1077 log.cc:924] Time spent T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260806 20:16:13.460714 30676 connection.cc:232] RPC call timeout handler was delayed by -1.2288380549871363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
I20260806 20:16:13.452116 30711 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
W20260806 20:16:13.476778 30676 connection.cc:232] RPC call timeout handler was delayed by -1.2288380549871363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
W20260806 20:16:13.477116 30676 connection.cc:232] RPC call timeout handler was delayed by -1.2261380550044123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15800000000000003s for a response
W20260806 20:16:13.477406 30676 connection.cc:232] RPC call timeout handler was delayed by -1.11363805498695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1705s for a response
W20260806 20:16:13.477699 30676 connection.cc:232] RPC call timeout handler was delayed by -1.0938380550069269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
I20260806 20:16:13.422197 1027 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:13.477788 32507 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:16:13.478124 30676 connection.cc:232] RPC call timeout handler was delayed by -1.0704380550014321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1753s for a response
W20260806 20:16:13.478452 30676 connection.cc:232] RPC call timeout handler was delayed by -1.0578380549850408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
W20260806 20:16:13.478765 30676 connection.cc:232] RPC call timeout handler was delayed by -1.0182380549958907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
I20260806 20:16:13.478329 32507 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.480334 30676 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.466701 1028 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:13.463980 1005 raft_consensus.cc:686] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:13.428896 812 consensus_queue.cc:1035] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260806 20:16:13.484308 32507 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:13.451033 838 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:13.496930 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:13.489817 1069 log.cc:924] Time spent T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260806 20:16:13.485931 1027 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.483209 879 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260806 20:16:13.491278 1070 log.cc:924] Time spent T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260806 20:16:13.491677 987 raft_consensus.cc:686] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:13.493393 972 consensus_queue.cc:1035] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:16:13.482992 30676 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.432166 1023 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.479321 961 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:13.472994 988 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:13.478904 690 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
W20260806 20:16:13.517481 30789 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1453) took 2113 ms (client timeout 1577 ms). Trace:
I20260806 20:16:13.519137 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.522082 1074 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:13.522456 1037 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:13.522908 1045 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:16:13.522856 961 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.524073 30757 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.524384 809 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:13.500478 877 consensus_queue.cc:1035] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260806 20:16:13.526127 31418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 510) took 1480 ms. Trace:
I20260806 20:16:13.584170 31418 rpcz_store.cc:276] 0806 20:16:12.045951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.050445 (+ 4494us) service_pool.cc:227] Handling call
0806 20:16:12.093104 (+ 42659us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:12.639962 (+546858us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:13.525884 (+885922us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b5400570b54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:13.526066 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1002241280}
I20260806 20:16:13.585827 961 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:13.586166 987 consensus_queue.cc:237] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.581131 667 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:13.589607 809 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:13.590134 809 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.590440 1005 consensus_queue.cc:237] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.592175 809 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:13.487773 810 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.526770 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:13.595935 30861 raft_consensus.cc:2379] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:13.528368 969 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:13.529577 1047 raft_consensus.cc:3032] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:13.530284 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:13.558799 30676 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:13.600162 30676 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:13.533911 30789 rpcz_store.cc:269] 0806 20:16:11.401996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:11.402132 (+ 136us) service_pool.cc:227] Handling call
0806 20:16:13.513205 (+2111073us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b54002df5d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:13.515322 (+ 2117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":946433792}
I20260806 20:16:13.502262 938 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:16:13.522572 935 consensus_meta.cc:287] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.536s user 0.003s sys 0.015s
I20260806 20:16:13.494499 608 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.520745 936 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:16:13.579537 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 55537.6us
I20260806 20:16:13.563871 690 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:13.583595 1080 log.cc:924] Time spent T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260806 20:16:13.601938 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:13.604858 30676 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:13.605080 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:13.578980 1069 log.cc:924] Time spent T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260806 20:16:13.607035 30757 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.608547 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.608348 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:13.562904 1076 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:13.621188 30870 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:13.580816 1023 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:13.628925 30783 raft_consensus.cc:1261] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:16:13.629115 30709 raft_consensus.cc:2379] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:13.630080 30757 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.632298 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.578276 871 raft_consensus.cc:686] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:13.606180 753 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:16:13.618925 1065 log.cc:924] Time spent T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.725s user 0.000s sys 0.004s
I20260806 20:16:13.603948 1047 raft_consensus.cc:729] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:13.643865 30757 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:13.613622 1030 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:13.501948 1002 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.614044 877 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:13.689390 30757 leader_election.cc:398] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Vote denied by peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735) with higher term. Message: Invalid argument: T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7.
I20260806 20:16:13.642976 936 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 20:16:13.688014 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 55798us
I20260806 20:16:13.731986 30676 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.733305 30676 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.734259 30676 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.735431 30676 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.736686 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.666632 1037 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:13.690131 1023 raft_consensus.cc:2726] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:13.569195 988 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.745129 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:13.644644 987 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:13.738238 1012 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:13.730361 936 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:13.739583 914 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:16:13.743561 1083 log.cc:924] Time spent T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.081s user 0.000s sys 0.002s
I20260806 20:16:13.739488 865 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:16:13.743965 1078 log.cc:924] Time spent T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.152s user 0.001s sys 0.001s
I20260806 20:16:13.690863 938 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:13.744269 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:13.744823 1090 raft_consensus.cc:2726] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:13.642462 810 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:13.751487 1074 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:13.670467 1068 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:13.745005 1081 log.cc:924] Time spent T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 20:16:13.722229 1066 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:13.752081 1063 consensus_queue.cc:1046] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260806 20:16:13.793889 1066 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.754227 1047 consensus_queue.cc:260] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.795413 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:13.796015 1066 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:13.738397 1076 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:13.796576 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:16:13.797089 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 745762us
I20260806 20:16:13.801682 608 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:13.756963 938 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:13.811781 938 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:13.812355 1059 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:13.812885 1059 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.813798 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:13.740558 1032 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:13.815083 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:13.816181 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:13.739159 877 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.744279 1043 raft_consensus.cc:2726] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:13.763284 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:13.839124 30868 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:13.766868 1089 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:13.841120 1089 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.845049 1029 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:13.844206 1089 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:13.848616 877 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:13.781793 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 674256us
I20260806 20:16:13.850607 988 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:13.853950 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1410) took 1566 ms. Trace:
I20260806 20:16:13.854362 30871 rpcz_store.cc:276] 0806 20:16:12.287497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:13.032364 (+744867us) service_pool.cc:227] Handling call
0806 20:16:13.032722 (+ 358us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:13.105647 (+ 72925us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:13.105677 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:13.121439 (+ 15762us) log.cc:842] Serialized 26 byte log entry
0806 20:16:13.598929 (+477490us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b180044b540. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e735345 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb 00007f71512f6063 00007f71512f6411 00007f71516435cf
0806 20:16:13.698899 (+ 99970us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:13.741234 (+ 42335us) log.cc:842] Serialized 14 byte log entry
0806 20:16:13.741464 (+ 230us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:13.741538 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:13.853651 (+112113us) raft_consensus.cc:1671] finished
0806 20:16:13.853759 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:13.853893 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78967,"prepare.queue_time_us":40941,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":89256,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":9091,"spinlock_wait_cycles":32171648,"thread_start_us":99731,"threads_started":1,"wal-append.queue_time_us":628922}
I20260806 20:16:13.855700 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:13.856737 30871 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 8. Current term is 9. Ops: [5.3-8.5]
I20260806 20:16:13.753997 936 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:13.666358 753 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:13.791785 30709 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 20:16:13.865028 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:13.797725 844 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 20:16:13.798457 1074 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.803197 30870 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:13.797130 30867 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:13.665868 935 consensus_queue.cc:1035] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260806 20:16:13.806059 30706 raft_consensus.cc:1226] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:16:13.784926 1027 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:13.810617 619 raft_consensus.cc:3032] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 20:16:13.876657 619 raft_consensus.cc:729] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:13.816934 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:13.877326 619 consensus_queue.cc:260] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.878993 619 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:16:13.742316 1002 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:13.815871 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:13.827374 30705 raft_consensus.cc:1261] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:13.833698 31389 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:13.836004 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:13.814978 1059 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:13.837329 869 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.889159 869 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:13.815038 1032 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:13.891785 1032 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:13.892401 30789 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260806 20:16:13.892710 30789 pending_rounds.cc:76] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Aborting all ops after (but not including) 7
I20260806 20:16:13.892834 30789 pending_rounds.cc:97] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260806 20:16:13.892966 30789 raft_consensus.cc:2866] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:13.815886 31419 raft_consensus.cc:1261] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:16:13.766573 1012 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:16:13.868839 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:13.873425 31417 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:13.895745 30788 raft_consensus.cc:1226] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 3. Current term is 4. Ops: []
I20260806 20:16:13.873991 30868 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:13.897364 30709 raft_consensus.cc:1261] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260806 20:16:13.897526 30871 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: [4.2-7.3]
I20260806 20:16:13.899012 30870 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:13.899292 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 65683.5us
I20260806 20:16:13.903174 31389 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:13.757495 871 consensus_queue.cc:237] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:13.911784 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:13.914409 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:13.915835 30791 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:13.916889 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:13.869870 753 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:13.919396 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:13.964773 30867 raft_consensus.cc:2454] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:13.928874 30788 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260806 20:16:13.947568 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 46009.9us
I20260806 20:16:13.899127 969 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:13.919067 30786 raft_consensus.cc:1203] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 5.3->[6.4-6.4] Dedup: 6.4->[]
I20260806 20:16:13.943714 30790 raft_consensus.cc:1226] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:14.011847 30786 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260806 20:16:13.974752 31414 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260806 20:16:13.967924 31389 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:14.014389 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 46580.8us
I20260806 20:16:13.926147 667 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:13.907289 31418 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260806 20:16:13.912953 944 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:16:13.967248 30785 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:14.021158 954 raft_consensus.cc:2781] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:14.021557 932 consensus_queue.cc:1035] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:16:14.021960 954 raft_consensus.cc:686] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:14.022912 954 consensus_queue.cc:237] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.025964 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:13.920711 1086 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.326s user 0.003s sys 0.000s
I20260806 20:16:14.026753 30790 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:14.030786 30789 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:14.031906 30599 catalog_manager.cc:5495] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 6 to 7, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 7 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:13.917438 1041 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:14.036653 30786 raft_consensus.cc:1226] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260806 20:16:14.016692 938 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.041917 30785 raft_consensus.cc:1226] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:13.904008 1028 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.045943 30792 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:14.054049 1037 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:14.057011 30791 raft_consensus.cc:1226] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:16:14.058212 938 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:14.040035 1074 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.011570 969 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:13.917200 924 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.127s
I20260806 20:16:14.086984 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:14.083369 30655 placement_policy_util.cc:244] tablet 5ba1969a3911423e8e51fc4f2962515d: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
I20260806 20:16:14.041471 1041 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:14.091612 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1425) took 1113 ms. Trace:
I20260806 20:16:14.048830 1028 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:14.093273 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:14.092072 30864 rpcz_store.cc:276] 0806 20:16:12.979512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:13.152034 (+172522us) service_pool.cc:227] Handling call
0806 20:16:13.162874 (+ 10840us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 20:16:13.382566 (+219692us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:13.382597 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 20:16:13.398274 (+ 15677us) log.cc:842] Serialized 78 byte log entry
0806 20:16:13.499906 (+101632us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:13.519912 (+ 20006us) log.cc:842] Serialized 14 byte log entry
0806 20:16:13.520257 (+ 345us) log.cc:842] Serialized 14 byte log entry
0806 20:16:13.521609 (+ 1352us) log.cc:842] Serialized 14 byte log entry
0806 20:16:13.538786 (+ 17177us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:13.538871 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:13.588761 (+ 49890us) spinlock_profiling.cc:243] Waited 6.19 ms on lock 0x7b18001ddbe0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e753638 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d 00007f71579ab204 00007f71579ab039 00007f714e735dc6
0806 20:16:13.840024 (+251263us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:14.091377 (+251353us) raft_consensus.cc:1671] finished
0806 20:16:14.091400 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:14.091549 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89180,"prepare.queue_time_us":200175,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":270898,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":46438,"spinlock_wait_cycles":58285056,"thread_start_us":60156,"threads_started":1,"wal-append.queue_time_us":172219}
I20260806 20:16:14.081213 969 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:14.072079 936 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:14.097890 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 66831.9us
I20260806 20:16:14.099254 1044 raft_consensus.cc:3032] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:14.099929 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1650) took 1896 ms. Trace:
I20260806 20:16:14.092249 1041 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:14.103320 30703 rpcz_store.cc:276] 0806 20:16:12.203166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.203334 (+ 168us) service_pool.cc:227] Handling call
0806 20:16:12.203685 (+ 351us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:12.216823 (+ 13138us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:12.216855 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:12.217377 (+ 522us) log.cc:842] Serialized 52 byte log entry
0806 20:16:13.926539 (+1709162us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b180042f160. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e735345 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb 00007f71512f6063 00007f71512f6411 00007f71516435cf
0806 20:16:14.033999 (+107460us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:14.034037 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:14.034104 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:14.099582 (+ 65478us) raft_consensus.cc:1671] finished
0806 20:16:14.099606 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:14.099865 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1999,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":15107,"spinlock_wait_cycles":4737280,"thread_start_us":107271,"threads_started":1,"wal-append.queue_time_us":1875603}
I20260806 20:16:14.107846 1044 raft_consensus.cc:729] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:14.112001 1044 consensus_queue.cc:260] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.113905 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:14.119936 31389 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:14.119727 1103 consensus_queue.cc:1046] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:16:14.120527 932 consensus_queue.cc:1035] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:16:14.113356 1044 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:14.122805 936 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.106408 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:14.130474 924 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:14.133752 1059 raft_consensus.cc:3032] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:16:14.137868 1059 raft_consensus.cc:729] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:14.138747 1059 consensus_queue.cc:260] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.140123 1059 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:14.140651 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:14.133869 924 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:14.142441 924 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:14.155737 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 400007us
I20260806 20:16:14.170130 944 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:16:14.175529 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:14.176362 30790 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:14.178362 30783 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:14.183434 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:14.188613 969 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.189790 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:14.190044 30835 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:14.190479 30786 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:14.191188 1025 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:14.191594 1025 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:14.191946 1025 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:14.192241 30757 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:14.193373 1091 log.cc:924] Time spent T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.618s user 0.001s sys 0.003s
I20260806 20:16:14.193543 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:14.193670 987 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.194296 30785 raft_consensus.cc:2379] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:14.194200 987 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.195013 30832 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:14.196410 987 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.189165 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:14.200053 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.200708 30832 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:14.201305 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.201872 30832 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:14.202386 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.202697 1027 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:14.203315 1027 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:14.170030 1041 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.203939 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:14.204341 30655 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 58d59beb597f4ed99d84d102ddb41ff5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:14.204828 1032 consensus_queue.cc:1046] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:14.205212 1041 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:14.206082 30709 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260806 20:16:14.207170 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:14.207561 1085 log.cc:924] Time spent T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.633s user 0.002s sys 0.000s
I20260806 20:16:14.209461 31418 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:14.210836 30832 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:14.203894 1027 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:14.212001 871 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:14.152669 924 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.215220 914 consensus_queue.cc:1046] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:14.216275 914 raft_consensus.cc:3032] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:14.216625 914 raft_consensus.cc:729] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:16:14.217592 986 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20260806 20:16:14.216275 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bf7304ca0290443b884297d0e8744607, TS d4181d4d907f477f9feddc44ba3b9207 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:14.219139 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet bf7304ca0290443b884297d0e8744607, TS d4181d4d907f477f9feddc44ba3b9207 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:14.220242 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 429381us
I20260806 20:16:14.220599 30832 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:14.205925 1032 raft_consensus.cc:3032] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:14.221685 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:14.133776 753 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:14.227746 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 365602us
I20260806 20:16:14.228137 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:14.228709 30787 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:14.229663 30706 raft_consensus.cc:1226] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:14.238252 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:14.230687 753 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.246430 30786 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:16:14.249033 30791 raft_consensus.cc:3032] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:14.249468 30791 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:14.260267 30712 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:14.261829 30865 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 209) took 1830 ms (client timeout 2065 ms). Trace:
I20260806 20:16:14.261929 30863 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
W20260806 20:16:14.262420 30865 rpcz_store.cc:269] 0806 20:16:12.431795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.698561 (+266766us) service_pool.cc:227] Handling call
0806 20:16:14.261612 (+1563051us) spinlock_profiling.cc:243] Waited 82.5 ms on lock 0x7b54000b0754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:14.261768 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":189794176}
I20260806 20:16:14.293166 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:14.294032 30709 raft_consensus.cc:1226] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:14.298332 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 66580us
I20260806 20:16:14.235913 1021 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:14.250418 30791 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.231897 1074 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:14.220185 924 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:14.300720 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:14.301952 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 67883.2us
W20260806 20:16:14.302515 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 43879.3us
I20260806 20:16:14.224043 1032 raft_consensus.cc:729] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:16:14.303402 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 59533.6us
I20260806 20:16:14.306602 30791 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:14.307657 30757 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:16:14.304051 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 43625.3us
I20260806 20:16:14.307672 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:14.308581 30707 raft_consensus.cc:1261] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 20:16:14.304220 30676 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:14.309834 30787 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:14.309278 30834 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:14.311955 30834 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:14.314000 810 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:14.311903 1066 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:14.315279 31389 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:14.316228 1041 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:14.316892 1041 raft_consensus.cc:686] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:14.317875 1041 consensus_queue.cc:237] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.324102 30834 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:14.325531 808 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:14.326996 30834 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:14.249185 973 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.408013 30787 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:14.409482 932 consensus_queue.cc:1046] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:16:14.410662 932 raft_consensus.cc:3032] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:14.411002 932 raft_consensus.cc:729] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:14.411837 932 consensus_queue.cc:260] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.413163 932 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:14.232774 935 consensus_queue.cc:1035] T 01fd0995e74a449090ea2b00062f625e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260806 20:16:14.333561 808 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:14.244357 987 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
I20260806 20:16:14.302508 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:14.250973 1068 consensus_queue.cc:1046] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
W20260806 20:16:14.426309 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 119318us
I20260806 20:16:14.223912 914 consensus_queue.cc:260] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.439136 31418 raft_consensus.cc:2379] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 9: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:14.419095 973 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.421819 1030 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:14.448776 987 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.308167 1028 consensus_queue.cc:1046] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.121s
I20260806 20:16:14.460572 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:14.474478 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 166900us
I20260806 20:16:14.475282 30708 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:14.494820 30757 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:14.521035 30791 raft_consensus.cc:1261] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260806 20:16:14.521677 30791 pending_rounds.cc:76] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Aborting all ops after (but not including) 4
I20260806 20:16:14.521972 30791 pending_rounds.cc:97] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260806 20:16:14.523757 30791 raft_consensus.cc:2866] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:14.417586 1101 consensus_queue.cc:1046] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.182s
I20260806 20:16:14.434404 1037 consensus_queue.cc:1035] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260806 20:16:14.560056 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:14.547971 1019 raft_consensus.cc:2781] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:14.561282 1019 raft_consensus.cc:686] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:14.562939 1019 consensus_queue.cc:237] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.566009 1101 raft_consensus.cc:3032] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:14.566627 1101 raft_consensus.cc:729] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:14.568120 973 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:14.539760 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 761e8328b9d247fcb1d00f0aeff373f0, TS 42cabca48db7403b8523a3382a046043 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:14.571837 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 761e8328b9d247fcb1d00f0aeff373f0, TS 42cabca48db7403b8523a3382a046043 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.467582 1030 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
W20260806 20:16:14.531002 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 63143.9us
I20260806 20:16:14.436944 1017 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.095s
I20260806 20:16:14.491744 869 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:14.526216 914 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:14.533901 1089 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.489952 987 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.575358 973 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:14.549997 1028 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.580482 1028 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.580924 1089 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:14.597229 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 179890us
I20260806 20:16:14.554273 969 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.575591 1030 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:14.632591 30835 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:14.635746 30787 raft_consensus.cc:3032] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:14.636955 30787 raft_consensus.cc:729] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:14.638017 30787 consensus_queue.cc:260] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.642534 30787 raft_consensus.cc:3037] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:14.615346 938 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:14.620896 871 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:14.628211 865 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:14.629508 869 raft_consensus.cc:686] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:16:14.659638 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 328546us
I20260806 20:16:14.667611 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:14.644349 1040 raft_consensus.cc:3032] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:14.674567 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:14.617298 973 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.648164 1062 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:16:14.705783 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 372425us
I20260806 20:16:14.706305 30787 raft_consensus.cc:1261] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260806 20:16:14.707154 30787 pending_rounds.cc:76] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Aborting all ops after (but not including) 3
I20260806 20:16:14.707410 30787 pending_rounds.cc:97] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 20:16:14.707661 30787 raft_consensus.cc:2866] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:14.657434 1021 raft_consensus.cc:686] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:14.669348 961 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 20:16:14.714521 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:14.681674 1066 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:14.756019 30869 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 6: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:14.757176 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:14.671391 871 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.772010 30864 raft_consensus.cc:1261] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 20:16:14.784902 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.785653 30710 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
W20260806 20:16:14.787343 1114 log.cc:924] Time spent T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20260806 20:16:14.787573 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:14.788455 1110 log.cc:924] Time spent T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.232s user 0.001s sys 0.000s
I20260806 20:16:14.710566 1041 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:14.789312 1041 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.789951 30709 raft_consensus.cc:1261] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 20:16:14.790201 1112 log.cc:924] Time spent T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.214s user 0.001s sys 0.000s
I20260806 20:16:14.788280 30711 raft_consensus.cc:1261] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:16:14.784291 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:14.670984 938 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.791100 30869 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:14.788766 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:14.783072 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:14.791872 869 consensus_queue.cc:237] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.674248 865 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.793593 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.793984 1021 consensus_queue.cc:237] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.775246 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.795382 865 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:14.792626 1041 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:14.796437 1041 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 20:16:14.798334 938 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:14.679507 1044 consensus_meta.cc:287] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.546s user 0.008s sys 0.008s
I20260806 20:16:14.774243 30868 raft_consensus.cc:1261] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:14.801396 30868 raft_consensus.cc:1261] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:16:14.803253 31420 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48918 (request call id 409) took 1828 ms. Trace:
I20260806 20:16:14.793632 1124 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:14.802898 30868 raft_consensus.cc:1226] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 20:16:14.804626 877 consensus_queue.cc:1035] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:14.805855 1028 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.806691 935 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.807422 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:14.803740 31420 rpcz_store.cc:276] 0806 20:16:12.975376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:12.975532 (+ 156us) service_pool.cc:227] Handling call
0806 20:16:12.975914 (+ 382us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:13.020519 (+ 44605us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:13.024153 (+ 3634us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:13.770986 (+746833us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b18004082e0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e6ca4ca 00007f7157d9ff99 00007f7155517bcf 00007f714e7537b6 00007f714e756b11 00007f714e756ac9 00007f714e756a59 00007f714e756a21 00007f714e755d1d
0806 20:16:14.145976 (+374990us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b2c00081bb8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:16:14.146476 (+ 500us) log.cc:842] Serialized 26 byte log entry
0806 20:16:14.419561 (+273085us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:14.421334 (+ 1773us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:14.426459 (+ 5125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:14.803021 (+376562us) raft_consensus.cc:1671] finished
0806 20:16:14.803040 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:14.803191 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40653,"raft.queue_time_us":28906,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":8689,"spinlock_wait_cycles":28819072,"thread_start_us":182752,"threads_started":1,"wal-append.queue_time_us":610520}
I20260806 20:16:14.807165 935 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.809520 987 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:14.808099 31414 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:14.807338 877 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.810750 877 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.811283 30754 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:14.811903 1061 raft_consensus.cc:3032] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:14.812304 1061 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:14.812417 1089 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.812714 1124 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.815111 1124 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.757501 30867 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: [4.5-6.6]
I20260806 20:16:14.813099 877 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:14.809836 935 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.825189 30599 catalog_manager.cc:5495] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:14.826732 30599 catalog_manager.cc:5495] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 4 to 6, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 6 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:14.827659 30603 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34006 (request call id 127) took 8299 ms. Trace:
I20260806 20:16:14.826570 30790 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:16:14.831395 690 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:14.832237 1009 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:16:14.829161 30599 catalog_manager.cc:5495] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 4 to 6, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 6 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
W20260806 20:16:14.834947 30861 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1530) took 1410 ms (client timeout 1539 ms). Trace:
W20260806 20:16:14.835409 30861 rpcz_store.cc:269] 0806 20:16:13.424309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:13.432753 (+ 8444us) service_pool.cc:227] Handling call
0806 20:16:14.834776 (+1402023us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b5400358254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:14.834892 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":373125376}
I20260806 20:16:14.835364 30599 catalog_manager.cc:5495] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:14.828042 30603 rpcz_store.cc:276] 0806 20:16:06.528710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:06.529201 (+ 491us) service_pool.cc:227] Handling call
0806 20:16:14.827591 (+8298390us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:16:08.733115 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:16:08.733672 (+ 557us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:16:08.733707 (+ 35us) write_op.cc:435] Acquired schema lock
0806 20:16:08.733719 (+ 12us) tablet.cc:660] Decoding operations
0806 20:16:08.734832 (+ 1113us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:16:08.734987 (+ 155us) write_op.cc:641] Partition lock acquired for write op
0806 20:16:08.735024 (+ 37us) tablet.cc:683] Acquiring locks for 13 operations
0806 20:16:08.736094 (+ 1070us) tablet.cc:699] Row locks acquired
0806 20:16:08.736114 (+ 20us) write_op.cc:260] PREPARE: finished
0806 20:16:08.736266 (+ 152us) write_op.cc:270] Start()
0806 20:16:08.736376 (+ 110us) write_op.cc:276] Timestamp: P: 1786047368736220 usec, L: 0
0806 20:16:08.736391 (+ 15us) op_driver.cc:331] REPLICATION: starting
0806 20:16:09.139534 (+403143us) log.cc:842] Serialized 5202 byte log entry
0806 20:16:10.221853 (+1082319us) op_driver.cc:444] REPLICATION: finished
0806 20:16:10.384524 (+162671us) write_op.cc:301] APPLY: starting
0806 20:16:10.384609 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0806 20:16:10.384882 (+ 273us) tablet.cc:1368] finished BulkCheckPresence
0806 20:16:10.385000 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:16:10.418950 (+ 33950us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:16:10.419246 (+ 296us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0806 20:16:10.419300 (+ 54us) write_op.cc:312] APPLY: finished
0806 20:16:10.419638 (+ 338us) log.cc:842] Serialized 95 byte log entry
0806 20:16:10.419824 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:10.419994 (+ 170us) write_op.cc:454] Released schema lock
0806 20:16:10.420049 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0806 20:16:10.420426 (+ 377us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:10.420453 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":37244,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":162373,"apply.run_cpu_time_us":4212,"apply.run_wall_time_us":36401,"num_ops":13,"prepare.queue_time_us":513251,"prepare.run_cpu_time_us":22399,"prepare.run_wall_time_us":970807,"raft.queue_time_us":160684,"raft.run_cpu_time_us":67523,"raft.run_wall_time_us":148713,"replication_time_us":1485362,"thread_start_us":757362,"threads_started":4,"wal-append.queue_time_us":676297,"wal-append.run_cpu_time_us":24077,"wal-append.run_wall_time_us":1547898}]]}
W20260806 20:16:14.810087 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 53608.4us
I20260806 20:16:14.841204 1064 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:14.841673 1064 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:14.842007 1064 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:14.843336 1025 consensus_meta.cc:287] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.651s user 0.014s sys 0.000s
I20260806 20:16:14.846434 30599 catalog_manager.cc:5495] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 4 to 6, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 6 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:14.820524 1032 consensus_queue.cc:260] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.757236 1012 consensus_queue.cc:1046] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260806 20:16:14.847656 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:14.818190 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1527) took 1401 ms. Trace:
I20260806 20:16:14.820231 871 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:14.771432 969 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.677392 1040 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:14.893428 30869 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
W20260806 20:16:14.892696 32507 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.033s sys 0.058s
I20260806 20:16:14.714074 1118 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:14.859483 30599 catalog_manager.cc:5495] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 4 to 6, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 6 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
W20260806 20:16:14.897153 32507 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s user 0.033s sys 0.058s
I20260806 20:16:14.897001 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1420) took 2467 ms. Trace:
I20260806 20:16:14.898484 30863 rpcz_store.cc:276] 0806 20:16:12.429691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:13.053933 (+624242us) service_pool.cc:227] Handling call
0806 20:16:14.261749 (+1207816us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:14.292801 (+ 31052us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:14.896831 (+604030us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54000b0750. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:14.896947 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2776950144}
I20260806 20:16:14.898999 30599 catalog_manager.cc:5495] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 1 to 5, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 5 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:14.754542 924 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:14.900736 30599 catalog_manager.cc:5495] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 1 to 5, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 5 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:14.893486 30866 rpcz_store.cc:276] 0806 20:16:13.416669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:13.842823 (+426154us) service_pool.cc:227] Handling call
0806 20:16:13.843172 (+ 349us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:14.171537 (+328365us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:16:14.171952 (+ 415us) log.cc:842] Serialized 14 byte log entry
0806 20:16:14.301990 (+130038us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:14.332072 (+ 30082us) log.cc:842] Serialized 52 byte log entry
0806 20:16:14.332214 (+ 142us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:14.332440 (+ 226us) log.cc:842] Serialized 14 byte log entry
0806 20:16:14.652532 (+320092us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:14.652624 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:14.817944 (+165320us) raft_consensus.cc:1671] finished
0806 20:16:14.817969 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:14.818121 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":338255,"prepare.queue_time_us":879,"raft.queue_time_us":427627,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":6029,"spinlock_wait_cycles":11342720,"thread_start_us":447614,"threads_started":2,"wal-append.queue_time_us":376636}
W20260806 20:16:14.910285 30673 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.019s sys 0.035s
W20260806 20:16:14.913882 30673 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.020s sys 0.035s
W20260806 20:16:14.914147 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 581175us
I20260806 20:16:14.914476 30599 catalog_manager.cc:5495] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 4 to 6, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 42cabca48db7403b8523a3382a046043 (127.28.186.193). New cstate: current_term: 6 leader_uuid: "42cabca48db7403b8523a3382a046043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:14.916844 30599 catalog_manager.cc:5495] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 reported cstate change: term changed from 4 to 6, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 6 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:14.856436 961 raft_consensus.cc:686] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:14.919759 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:14.902146 1122 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.709359 1119 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.210s
W20260806 20:16:14.930756 932 consensus_meta.cc:287] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.517s user 0.009s sys 0.003s
I20260806 20:16:14.930542 1122 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.925030 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:14.931452 1012 raft_consensus.cc:3032] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:16:14.931824 1012 raft_consensus.cc:729] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:14.933063 30709 raft_consensus.cc:1226] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260806 20:16:14.926542 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:14.928449 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.939190 30706 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 6
I20260806 20:16:14.929725 1068 consensus_queue.cc:1035] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:16:14.924114 30756 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:14.711915 1062 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.895310 981 consensus_queue.cc:1046] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260806 20:16:14.924763 1040 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.935467 30870 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:14.709180 1101 consensus_queue.cc:260] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.932569 1012 consensus_queue.cc:260] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.864936 1025 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.903976 969 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:14.986726 31389 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:14.990840 30757 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:14.992795 30706 raft_consensus.cc:1261] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:16:14.911547 924 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.947078 973 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:14.948292 1100 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:14.996727 31389 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:14.920614 999 consensus_queue.cc:1046] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260806 20:16:14.813212 1061 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:14.949828 1045 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:15.002713 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.994268 1119 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:15.005951 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.904877 1118 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.031255 30710 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:15.032162 1064 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:15.000559 932 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:14.997921 871 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:14.997548 942 consensus_queue.cc:1035] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.038s
I20260806 20:16:15.001026 1101 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:15.035935 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:14.999178 1037 raft_consensus.cc:2726] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:15.040158 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:15.040565 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:15.049870 30710 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:15.004506 1100 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:15.005877 1040 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:15.004997 1012 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:15.005672 844 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:15.046795 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:15.047019 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:15.035045 999 raft_consensus.cc:3032] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:15.052516 30864 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:15.053251 30709 raft_consensus.cc:2379] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:15.054201 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:14.984261 961 consensus_queue.cc:237] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.005475 838 consensus_queue.cc:1035] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260806 20:16:15.029415 1045 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.056401 1100 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:15.080796 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:15.081486 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:16:15.098750 30834 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:15.036237 1119 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.105597 30866 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:16:15.108690 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:15.110116 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:15.110548 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:15.119424 30789 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 2.
I20260806 20:16:15.120451 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:15.118865 30783 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:15.126677 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:15.126873 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:15.127424 31423 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:16:15.067514 898 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:15.058362 999 raft_consensus.cc:729] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:15.132293 30834 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:15.132429 30864 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:15.133561 30834 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:15.130811 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:15.135510 30789 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:15.137589 30832 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:15.050315 871 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.138262 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:15.138873 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 276229us
I20260806 20:16:15.138778 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:15.156160 30783 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:15.156615 30785 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:16:15.158079 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:15.155537 30834 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:15.081314 1122 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:15.159531 30835 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:15.121459 1117 log.cc:924] Time spent T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20260806 20:16:15.162485 30676 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:15.138381 1121 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
I20260806 20:16:15.169589 30835 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:15.137646 988 raft_consensus.cc:2781] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:15.172916 30835 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:15.174190 30835 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:15.176465 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 185719us
I20260806 20:16:15.154246 1029 consensus_queue.cc:1035] T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260806 20:16:15.178889 31389 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:15.181761 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:15.160456 957 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:15.185029 30835 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:15.163811 935 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:15.166661 1034 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:15.137575 831 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:16:15.195276 30676 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.194:42029 timed out after 1.577s (SENT) [suppressed 18 similar messages]
I20260806 20:16:15.195864 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:15.196416 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:15.197135 30868 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer 42cabca48db7403b8523a3382a046043: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:16:15.198526 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:15.166476 1090 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:15.202879 30866 raft_consensus.cc:1226] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260806 20:16:15.174979 885 raft_consensus.cc:2781] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 20:16:15.190061 1115 log.cc:924] Time spent T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
I20260806 20:16:15.177642 988 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:15.236800 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:15.166780 936 consensus_queue.cc:1046] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260806 20:16:15.183779 690 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:16:15.154765 898 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.184469 1025 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:15.202032 808 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:15.182932 877 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:15.185158 1062 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:15.214632 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:15.247264 988 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:15.358210 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 108483us
W20260806 20:16:15.232277 1128 log.cc:924] Time spent T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.238s user 0.001s sys 0.000s
I20260806 20:16:15.191660 935 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:15.210608 1041 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:15.193480 969 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:15.204811 1034 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:15.372398 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 339) took 1257 ms. Trace:
I20260806 20:16:15.165616 999 consensus_queue.cc:260] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.233554 1124 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:15.242462 30861 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
W20260806 20:16:15.401945 1111 log.cc:924] Time spent T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20260806 20:16:15.428414 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:15.390899 1064 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:15.236079 1134 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:15.360209 1030 consensus_meta.cc:287] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.719s user 0.010s sys 0.000s
I20260806 20:16:15.164094 1132 consensus_queue.cc:1035] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20260806 20:16:15.194254 1037 consensus_queue.cc:1035] T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.178s
W20260806 20:16:15.247012 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 62079.2us
I20260806 20:16:15.441639 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:15.404331 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:15.393784 877 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:15.410630 30713 raft_consensus.cc:3032] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:15.421905 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:15.423437 30703 rpcz_store.cc:276] 0806 20:16:14.114532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:14.117870 (+ 3338us) service_pool.cc:227] Handling call
0806 20:16:14.118243 (+ 373us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:14.235391 (+117148us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:14.235421 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:14.245613 (+ 10192us) log.cc:842] Serialized 26 byte log entry
0806 20:16:14.710525 (+464912us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:14.710570 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:14.710641 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:15.372170 (+661529us) raft_consensus.cc:1671] finished
0806 20:16:15.372189 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:15.372338 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":126165,"raft.queue_time_us":124741,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":44176,"thread_start_us":293547,"threads_started":1,"wal-append.queue_time_us":614738}
W20260806 20:16:15.422931 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 176335us
I20260806 20:16:15.238021 885 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:15.451153 885 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:16:15.429667 30792 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.014s sys 0.005s
I20260806 20:16:15.393885 1131 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:15.431533 30861 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:15.393548 936 consensus_queue.cc:1225] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer 58d59beb597f4ed99d84d102ddb41ff5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:16:15.237141 831 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.405372 1118 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:15.239240 1090 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.395133 924 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:15.417928 935 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:15.448654 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:15.432276 1123 log.cc:924] Time spent T b7665c0d8832432f8948e82d823498fd P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.383s user 0.001s sys 0.000s
I20260806 20:16:15.469285 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:15.504163 1146 log.cc:924] Time spent T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260806 20:16:15.401947 1142 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20260806 20:16:15.429275 1034 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:15.425734 1041 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
W20260806 20:16:15.461054 30792 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.015s sys 0.006s
I20260806 20:16:15.428016 812 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:15.464831 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:15.460125 877 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:15.449786 30713 raft_consensus.cc:729] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:16:15.497141 914 consensus_meta.cc:287] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.918s user 0.010s sys 0.000s
I20260806 20:16:15.509192 30783 raft_consensus.cc:2379] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:15.508524 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:15.496315 1141 log.cc:924] Time spent T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.313s user 0.000s sys 0.001s
I20260806 20:16:15.518208 812 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.527928 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:15.534070 30789 raft_consensus.cc:2379] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 8: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:15.534802 30789 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:15.535360 30789 raft_consensus.cc:595] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:16:15.529755 898 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:15.514456 1041 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:15.508996 871 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:15.530683 831 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:15.531790 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:15.538846 30783 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:15.542240 30789 raft_consensus.cc:1226] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 20:16:15.490151 1051 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:15.504981 1045 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:15.537992 936 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:15.550140 1159 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:15.550742 1090 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:15.570188 936 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.560694 877 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.572887 877 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:15.566931 932 consensus_meta.cc:287] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.526s user 0.007s sys 0.003s
I20260806 20:16:15.573797 932 raft_consensus.cc:2726] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:16:15.545248 1133 log.cc:924] Time spent T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.338s user 0.000s sys 0.001s
I20260806 20:16:15.554466 1047 raft_consensus.cc:3037] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:15.577003 1009 raft_consensus.cc:3032] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:15.442540 1030 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.570298 1159 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.584550 1051 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.577416 1009 raft_consensus.cc:729] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:15.591532 1051 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:15.591638 1159 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:15.561282 1034 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:15.583778 885 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:15.579748 1100 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:15.603367 1150 log.cc:924] Time spent T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260806 20:16:15.591816 1009 consensus_queue.cc:260] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:15.570214 1136 log.cc:924] Time spent T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20260806 20:16:15.548841 1119 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:15.518784 30713 consensus_queue.cc:260] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.558425 30863 raft_consensus.cc:1226] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:15.561712 30866 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:15.599268 812 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:15.608481 885 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:15.722313 1034 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:15.699497 31418 consensus_meta.cc:287] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.766s user 0.009s sys 0.005s
I20260806 20:16:15.723786 31418 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
W20260806 20:16:15.616680 1012 consensus_meta.cc:287] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.556s user 0.007s sys 0.003s
I20260806 20:16:15.619980 1134 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:15.629571 1137 raft_consensus.cc:3032] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:15.628271 1061 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:15.621405 935 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.626374 1032 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:15.630088 1100 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:15.718627 1030 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:15.688019 30792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 296) took 1514 ms. Trace:
I20260806 20:16:15.678424 30713 raft_consensus.cc:3037] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:15.694340 30787 raft_consensus.cc:1226] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
W20260806 20:16:15.638854 1101 consensus_meta.cc:287] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.587s user 0.008s sys 0.001s
I20260806 20:16:15.705838 30791 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:16:15.775235 1156 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:15.678491 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:15.627554 1040 consensus_meta.cc:287] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.569s user 0.007s sys 0.001s
I20260806 20:16:15.785748 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:15.777431 1038 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:15.781703 1076 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:15.782694 1119 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:15.776906 1047 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:15.625733 1131 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:16:15.805003 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 92852.7us
I20260806 20:16:15.642292 1124 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:15.893239 30712 raft_consensus.cc:2396] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260806 20:16:15.905107 30710 raft_consensus.cc:2396] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 20:16:15.907702 1137 raft_consensus.cc:729] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:15.909550 1137 consensus_queue.cc:260] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.911867 1137 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:15.913578 30790 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:15.914335 987 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:15.915227 987 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:15.917593 987 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:15.808504 30792 rpcz_store.cc:276] 0806 20:16:14.173399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:14.173853 (+ 454us) service_pool.cc:227] Handling call
0806 20:16:14.174223 (+ 370us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:14.174588 (+ 365us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:16:14.195233 (+ 20645us) log.cc:842] Serialized 14 byte log entry
0806 20:16:14.816323 (+621090us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b18003c56e0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e735345 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb 00007f71512f6063 00007f71512f657b 00007f715167988a
0806 20:16:15.511749 (+695426us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:15.512172 (+ 423us) log.cc:842] Serialized 26 byte log entry
0806 20:16:15.512308 (+ 136us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:15.512335 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:15.512396 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:15.686804 (+174408us) raft_consensus.cc:1671] finished
0806 20:16:15.687492 (+ 688us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:15.687773 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47625,"prepare.queue_time_us":20940,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":1278,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":25769,"spinlock_wait_cycles":3360000,"thread_start_us":695221,"threads_started":1,"wal-append.queue_time_us":1362155}
I20260806 20:16:15.918087 1047 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:15.919596 30868 raft_consensus.cc:1226] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:15.920944 1047 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:15.921619 812 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:15.921851 1119 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.922467 812 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.924371 1119 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:15.924772 812 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:15.925714 30708 raft_consensus.cc:2396] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260806 20:16:15.925495 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 51831.7us
I20260806 20:16:15.925882 942 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:15.949975 30871 raft_consensus.cc:2396] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260806 20:16:15.902498 1043 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:15.959884 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:15.925724 943 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:15.995895 30790 raft_consensus.cc:595] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:16:15.996428 30790 raft_consensus.cc:603] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:16:15.950695 942 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:15.921656 1156 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:15.997872 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:16.008976 30757 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.019s sys 0.030s
W20260806 20:16:16.010555 30757 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.810s user 0.019s sys 0.031s
I20260806 20:16:16.013742 812 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:16.014756 812 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:16.015668 1101 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:16.016484 1101 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:16.018924 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:16.019133 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:16.019892 31414 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:16.003089 30676 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.011s sys 0.025s
I20260806 20:16:16.022110 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:16.022809 31420 raft_consensus.cc:1261] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 20:16:16.024537 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:16.025435 31415 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:16:16.026058 31418 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 526) took 1473 ms (client timeout 1904 ms). Trace:
W20260806 20:16:16.026558 31418 rpcz_store.cc:269] 0806 20:16:14.552721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:14.559741 (+ 7020us) service_pool.cc:227] Handling call
0806 20:16:16.025752 (+1466011us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b5400501754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:16.025944 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":855608192}
I20260806 20:16:16.027678 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:15.969558 1038 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:16.030663 999 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:16.031683 31423 raft_consensus.cc:1226] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:16.032393 1038 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:16.034650 30676 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.666s user 0.012s sys 0.025s
W20260806 20:16:16.035168 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 839865us
I20260806 20:16:16.017325 812 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:16.038499 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:16.039978 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:16.036029 30712 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
W20260806 20:16:16.004943 31387 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.008s sys 0.030s
I20260806 20:16:16.019253 1101 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:16.041419 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:16.050262 31387 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.008s sys 0.030s
I20260806 20:16:15.959556 1076 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:16.012666 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 811927us
I20260806 20:16:15.974758 987 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:16.039971 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:16.071776 30706 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.817s user 0.014s sys 0.022s
I20260806 20:16:16.002693 1043 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:16.013778 1131 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:16.102917 844 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:16.104363 1156 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:16.068660 943 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:16.106001 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33214 (request call id 623) took 1531 ms. Trace:
I20260806 20:16:16.106402 30788 rpcz_store.cc:276] 0806 20:16:14.574800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:14.591605 (+ 16805us) service_pool.cc:227] Handling call
0806 20:16:16.036486 (+1444881us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54001d10d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4af0 00007f7151677f2a 00007f71565b4c6e 00007f714fcb7807 00007f714fcb7792 00007f714fcb76c1 00007f714fcb763c 00007f714fcb68f2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:16.105810 (+ 69324us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3316376960}
I20260806 20:16:16.106735 943 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:16.109668 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 218) took 1370 ms. Trace:
I20260806 20:16:16.110011 30867 raft_consensus.cc:1226] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260806 20:16:16.111032 1019 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.133s user 0.027s sys 0.032s
W20260806 20:16:16.111430 1019 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.182s user 0.027s sys 0.032s
I20260806 20:16:16.117419 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:16:16.112797 30709 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.028s sys 0.012s
W20260806 20:16:16.123652 30706 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.954s user 0.014s sys 0.024s
I20260806 20:16:16.112416 30712 raft_consensus.cc:1226] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:16.133394 31415 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:16:16.134331 31419 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:16.135017 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
W20260806 20:16:16.128345 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 49793.5us
W20260806 20:16:16.134656 30709 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s user 0.029s sys 0.013s
I20260806 20:16:16.124694 1012 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:16.155663 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:16.110098 30865 rpcz_store.cc:276] 0806 20:16:14.712399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:14.712514 (+ 115us) service_pool.cc:227] Handling call
0806 20:16:14.712762 (+ 248us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:15.061977 (+349215us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:15.062005 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:15.117695 (+ 55690us) log.cc:842] Serialized 26 byte log entry
0806 20:16:15.371594 (+253899us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:15.410685 (+ 39091us) log.cc:842] Serialized 14 byte log entry
0806 20:16:16.000673 (+589988us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:16.062734 (+ 62061us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:16.073235 (+ 10501us) raft_consensus.cc:1671] finished
0806 20:16:16.073253 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:16.082774 (+ 9521us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22252,"prepare.queue_time_us":38854,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":125437,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"spinlock_wait_cycles":39321472,"thread_start_us":1045696,"threads_started":3,"wal-append.queue_time_us":408985}
I20260806 20:16:16.158980 31416 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:16.128386 1101 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:16.165515 30711 raft_consensus.cc:1261] T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 20:16:16.167263 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:16.167511 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 308) took 1223 ms. Trace:
W20260806 20:16:16.167616 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 46679.4us
I20260806 20:16:16.169356 30787 rpcz_store.cc:276] 0806 20:16:14.944102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:14.944302 (+ 200us) service_pool.cc:227] Handling call
0806 20:16:14.944576 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:15.830011 (+885435us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:16.167206 (+337195us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b54003b24d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:16.167440 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1722662784}
W20260806 20:16:16.156194 1125 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.964s user 0.009s sys 0.033s
I20260806 20:16:16.170794 30754 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:16.143482 987 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:16.150344 935 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:16.150944 1131 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:16.185606 987 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:16.186563 1131 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:16.187731 1100 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:16.188362 30676 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:16.188567 1100 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:16.189975 30757 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:16.190251 1120 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:16.190761 1120 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:16.191212 1100 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:16.193404 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f0cdb10a40ac4d0bbcfc4ed368b6dc43, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:16.193871 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f0cdb10a40ac4d0bbcfc4ed368b6dc43, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:16.194901 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:16.197041 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:16.203532 1101 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:16.205536 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1540) took 1009 ms. Trace:
I20260806 20:16:16.196972 1074 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:16.218760 30784 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 9 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260806 20:16:16.197780 1125 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.103s user 0.009s sys 0.034s
W20260806 20:16:16.206663 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 187876us
I20260806 20:16:16.193723 1126 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:16.175427 1019 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:16.218398 30867 rpcz_store.cc:276] 0806 20:16:15.196037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:15.196916 (+ 879us) service_pool.cc:227] Handling call
0806 20:16:15.204301 (+ 7385us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:16.110545 (+906244us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:16.205206 (+ 94661us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b540019cb54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:16.205469 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2014143360}
I20260806 20:16:16.232931 31414 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:16.193421 1120 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:16.233389 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1700) took 1214 ms. Trace:
I20260806 20:16:16.167287 1051 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:16.187546 1012 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:16.795756 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 614182us
I20260806 20:16:16.480937 31416 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
W20260806 20:16:16.789520 30869 consensus_meta.cc:287] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.569s user 0.009s sys 0.000s
W20260806 20:16:16.764870 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 68352.3us
W20260806 20:16:16.495777 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 334103us
I20260806 20:16:16.498481 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 305) took 1668 ms. Trace:
I20260806 20:16:16.549489 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1701) took 1547 ms. Trace:
I20260806 20:16:16.359814 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:16.480549 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 105789us
I20260806 20:16:16.493673 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1467) took 2579 ms. Trace:
I20260806 20:16:16.550280 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1709) took 1109 ms. Trace:
I20260806 20:16:16.816659 30869 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:16.330833 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:16.696691 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:16.341150 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:16.835776 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1537) took 1629 ms (client timeout 1602 ms). Trace:
I20260806 20:16:16.824354 30784 rpcz_store.cc:276] 0806 20:16:13.914420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:13.914560 (+ 140us) service_pool.cc:227] Handling call
0806 20:16:13.919831 (+ 5271us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:16.219492 (+2299661us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:16.490617 (+271125us) spinlock_profiling.cc:243] Waited 2.29 s on lock 0x7b540026eb54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:16.493570 (+ 2953us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5257502080}
I20260806 20:16:16.824587 30710 rpcz_store.cc:276] 0806 20:16:15.441183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:15.441328 (+ 145us) service_pool.cc:227] Handling call
0806 20:16:16.550022 (+1108694us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b5400211bd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:16.550202 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":787258240}
W20260806 20:16:16.544557 1178 log.cc:924] Time spent T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
W20260806 20:16:16.368041 1174 log.cc:924] Time spent T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260806 20:16:16.856513 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:16.859231 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:16.823163 30791 rpcz_store.cc:276] 0806 20:16:14.829674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:14.829825 (+ 151us) service_pool.cc:227] Handling call
0806 20:16:14.830126 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:15.831236 (+1001110us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:16.498194 (+666958us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b54003b29d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:16.498412 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1970578176}
I20260806 20:16:16.228925 1167 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:16.596033 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 91051.5us
I20260806 20:16:16.228952 1062 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 20:16:16.837668 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:16.836674 30869 rpcz_store.cc:269] 0806 20:16:15.195269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.141536 (+946267us) service_pool.cc:227] Handling call
0806 20:16:16.824812 (+683276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:16.823513 30708 rpcz_store.cc:276] 0806 20:16:15.002285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:15.002442 (+ 157us) service_pool.cc:227] Handling call
0806 20:16:16.549194 (+1546752us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b54001b5154. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:16.549382 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2104547200}
I20260806 20:16:16.793895 944 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:16.792620 1009 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:16.887655 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
W20260806 20:16:16.548606 1183 log.cc:924] Time spent T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.142s user 0.001s sys 0.001s
W20260806 20:16:16.273741 1179 log.cc:924] Time spent T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20260806 20:16:16.497169 936 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:16.278189 1100 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:16.598141 942 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:16.331698 1166 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:16:16.342566 1043 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:16.899545 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1449) took 1834 ms. Trace:
I20260806 20:16:16.331463 865 consensus_queue.cc:1046] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.685s
I20260806 20:16:16.905354 30783 raft_consensus.cc:2379] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:16.902877 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:16.905544 30792 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:16.244522 30706 rpcz_store.cc:276] 0806 20:16:14.997807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:14.997965 (+ 158us) service_pool.cc:227] Handling call
0806 20:16:14.998291 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:14.998483 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 20:16:15.167311 (+168828us) log.cc:842] Serialized 14 byte log entry
0806 20:16:16.152167 (+984856us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:16.170047 (+ 17880us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:16.174832 (+ 4785us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:16.183497 (+ 8665us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:16.211761 (+ 28264us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24704,"prepare.queue_time_us":60501,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":4396,"raft.queue_time_us":16053,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":22541,"spinlock_wait_cycles":6648704,"thread_start_us":1088058,"threads_started":2,"wal-append.queue_time_us":1153025}
I20260806 20:16:16.491765 1076 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:16.228926 1074 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:16.909540 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 492068us
I20260806 20:16:16.787731 506 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:16.273967 969 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:16.782845 1155 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:16.905478 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:16.910627 30870 rpcz_store.cc:276] 0806 20:16:15.065040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:15.065348 (+ 308us) service_pool.cc:227] Handling call
0806 20:16:15.115911 (+ 50563us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:15.161251 (+ 45340us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:15.161283 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:15.161320 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:15.161356 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:16.855670 (+1694314us) spinlock_profiling.cc:243] Waited 45.1 ms on lock 0x7b540019d054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:16.885856 (+ 30186us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:16.899337 (+ 13481us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":103830272}
I20260806 20:16:16.915457 30757 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:16.786520 1038 consensus_queue.cc:1035] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.118s
W20260806 20:16:16.918162 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 729908us
I20260806 20:16:16.918984 1007 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:16.919407 936 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:16.919808 1074 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:16.919850 753 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.010s sys 0.036s
I20260806 20:16:16.971498 30676 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:16.920529 30789 raft_consensus.cc:2396] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260806 20:16:16.922591 30868 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:16:16.923236 30707 raft_consensus.cc:1226] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:16.924039 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 354) took 1748 ms. Trace:
I20260806 20:16:16.925793 935 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:16.902336 1051 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:16.900815 1167 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:16.927105 30788 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:16.927116 1163 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:16.928118 1166 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:16.928632 1012 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:16.908376 944 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:16.915110 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:16.929273 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:16.944500 1043 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:16.929924 1124 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:16.931484 30832 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:16.932225 810 consensus_queue.cc:1035] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:16:16.934403 30711 raft_consensus.cc:1261] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260806 20:16:16.935075 1041 consensus_meta.cc:287] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 1.370s user 0.015s sys 0.000s
W20260806 20:16:16.936250 988 consensus_meta.cc:287] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.510s user 0.007s sys 0.010s
I20260806 20:16:17.003978 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:16.322096 1019 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:16.947973 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:16.945531 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1547) took 1393 ms. Trace:
I20260806 20:16:16.948558 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:16.948789 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:16.938261 1062 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:16.965217 30787 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:16.972692 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:16.976231 30754 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:16.973197 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:16.918195 667 consensus_queue.cc:1046] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.199s
W20260806 20:16:16.974902 753 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.010s sys 0.037s
I20260806 20:16:16.972934 30709 rpcz_store.cc:276] 0806 20:16:15.175264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:15.175515 (+ 251us) service_pool.cc:227] Handling call
0806 20:16:15.175876 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:15.176296 (+ 420us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:15.176324 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:15.183372 (+ 7048us) log.cc:842] Serialized 26 byte log entry
0806 20:16:16.184045 (+1000673us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:16.184121 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:16.184239 (+ 118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:16.923814 (+739575us) raft_consensus.cc:1671] finished
0806 20:16:16.923837 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:16.923977 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":97,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":23633,"thread_start_us":904273,"threads_started":1,"wal-append.queue_time_us":981056}
W20260806 20:16:16.978768 1137 consensus_meta.cc:287] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.066s user 0.013s sys 0.005s
I20260806 20:16:16.943234 1076 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:16:16.986510 30673 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.060s sys 0.067s
W20260806 20:16:16.975869 986 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.016s sys 0.041s
I20260806 20:16:16.971904 942 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.012758 988 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:17.023293 986 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.837s user 0.016s sys 0.042s
I20260806 20:16:17.026407 1165 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:16:16.990525 30834 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:17.003666 30784 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:16.332196 1126 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.028373 988 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:17.029436 1181 log.cc:924] Time spent T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.751s user 0.000s sys 0.003s
I20260806 20:16:16.977300 936 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:16.989490 999 consensus_meta.cc:287] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.958s user 0.003s sys 0.011s
I20260806 20:16:16.982995 878 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.011861 31416 raft_consensus.cc:2379] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 6: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:16.993208 1101 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:16.980985 1156 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
W20260806 20:16:17.021250 30673 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.778s user 0.060s sys 0.068s
I20260806 20:16:16.975525 969 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:17.043614 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 825023us
I20260806 20:16:17.044091 30783 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
W20260806 20:16:17.025097 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 93849us
I20260806 20:16:17.007324 30861 rpcz_store.cc:276] 0806 20:16:15.199124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:15.465969 (+266845us) service_pool.cc:227] Handling call
0806 20:16:15.466273 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:15.466472 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:16:15.466712 (+ 240us) log.cc:842] Serialized 14 byte log entry
0806 20:16:16.047315 (+580603us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b18000e51e0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e735345 00007f7157560a39 00007f714e7528bb 00007f714e750cbf 00007f714e752e22 00007f71512f05bb 00007f71512f6063 00007f71512f657b 00007f715167988a
0806 20:16:16.261001 (+213686us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:16.393318 (+132317us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:16.560412 (+167094us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:16.587653 (+ 27241us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:16.592845 (+ 5192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":208823,"prepare.queue_time_us":89,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":16190,"raft.queue_time_us":130373,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":70841,"spinlock_wait_cycles":9295744,"thread_start_us":73137,"threads_started":1,"wal-append.queue_time_us":793175}
I20260806 20:16:17.024560 1007 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.008755 935 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:16.991905 1166 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.035795 878 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:17.036115 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 56825.8us
I20260806 20:16:17.004747 1163 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.036796 961 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:17.026185 942 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.010092 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:16.972426 506 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.038774 30790 raft_consensus.cc:2379] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 9: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:17.019620 1041 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.041643 1172 raft_consensus.cc:2781] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:16:17.011070 1012 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:17.042892 1126 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.048420 30832 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:16:17.055001 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 83687.9us
I20260806 20:16:16.990937 1155 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.020520 1043 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:17.049911 1185 log.cc:924] Time spent T c0005c7c8586414a81bd94a504a598c9 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.683s user 0.005s sys 0.000s
I20260806 20:16:17.063561 1165 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:16:17.067479 1165 consensus_queue.cc:1225] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer d4181d4d907f477f9feddc44ba3b9207 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 20:16:17.067523 31417 raft_consensus.cc:1226] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:17.068202 935 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:17.069990 878 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:17.073566 30835 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:17.043045 1060 consensus_queue.cc:1046] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260806 20:16:17.068130 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1710) took 1584 ms. Trace:
I20260806 20:16:17.141516 1120 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.142278 1120 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.143141 1163 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.144610 1120 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.144660 1167 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:17.050241 1156 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.148052 1043 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.148162 1156 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.148665 1043 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:17.054071 961 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:16:17.149715 1177 log.cc:924] Time spent T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260806 20:16:17.047416 1124 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.150568 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:17.151381 1120 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:17.151815 1124 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.153051 30676 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:17.153548 506 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.080930 30703 rpcz_store.cc:276] 0806 20:16:15.461933 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:15.462113 (+ 180us) service_pool.cc:227] Handling call
0806 20:16:15.462485 (+ 372us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:15.465992 (+ 3507us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:15.466021 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:15.504157 (+ 38136us) log.cc:842] Serialized 26 byte log entry
0806 20:16:15.908053 (+403896us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:15.908165 (+ 112us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:15.908262 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:17.045957 (+1137695us) raft_consensus.cc:1671] finished
0806 20:16:17.045979 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:17.046152 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3151,"raft.queue_time_us":32802,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":403138,"threads_started":1,"wal-append.queue_time_us":692764}
I20260806 20:16:17.064936 999 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:17.149653 961 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:17.166896 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 81585.3us
I20260806 20:16:17.145807 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.169777 31421 raft_consensus.cc:2379] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 9: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
W20260806 20:16:17.166857 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 119240us
I20260806 20:16:17.066560 1038 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.179265 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:17.180681 30784 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:17.178131 30834 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:17.052640 1076 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.215641 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.220101 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.178604 1120 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.213068 878 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:17.174468 1172 raft_consensus.cc:686] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:17.182992 1074 raft_consensus.cc:2741] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260806 20:16:17.043895 1062 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.227487 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:17.230753 31414 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:17.234262 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:17.215009 1126 raft_consensus.cc:2741] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:16:17.203833 969 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.218786 944 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.247973 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:17.247973 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.249311 30707 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:17.249372 878 raft_consensus.cc:2726] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:17.221145 1163 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:17.228260 1125 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:17.250569 1125 raft_consensus.cc:686] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:17.250329 1163 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.251579 1125 consensus_queue.cc:237] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.223106 1012 consensus_queue.cc:1046] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260806 20:16:17.253470 1163 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.253904 1166 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.232765 1120 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.256815 1120 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:17.261600 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 86417.7us
I20260806 20:16:17.228444 999 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:17.313017 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 160061us
W20260806 20:16:17.315742 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 267135us
I20260806 20:16:17.318923 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:17.259582 1163 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.307921 1019 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:17.322897 1186 log.cc:924] Time spent T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.999s user 0.003s sys 0.000s
I20260806 20:16:17.228021 1038 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.322321 936 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.340421 30785 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:17.319391 1155 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:17.343099 1180 log.cc:924] Time spent T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 1.020s user 0.000s sys 0.006s
I20260806 20:16:17.353168 30835 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:17.344574 969 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:17.360800 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 48064.7us
I20260806 20:16:17.331806 1163 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.340471 1166 consensus_queue.cc:1046] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.017s
I20260806 20:16:17.359524 1100 raft_consensus.cc:686] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:17.345556 1172 consensus_queue.cc:237] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.389655 30787 raft_consensus.cc:2454] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:17.390465 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1935) took 1066 ms. Trace:
I20260806 20:16:17.390853 30787 rpcz_store.cc:276] 0806 20:16:16.324022 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.858191 (+534169us) service_pool.cc:227] Handling call
0806 20:16:17.390394 (+532203us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:17.380000 943 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.031s user 0.002s sys 0.025s
I20260806 20:16:17.360139 969 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.378063 1167 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:17.379231 1062 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:17.403229 943 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.054s user 0.002s sys 0.025s
W20260806 20:16:17.412922 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 443665us
W20260806 20:16:17.413988 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 72235us
I20260806 20:16:17.416018 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.404044 999 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:17.433311 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8ec60d6f5b6c486b80781ad3cb85a06b, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:17.433751 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 8ec60d6f5b6c486b80781ad3cb85a06b, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.437975 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.440017 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.445801 1076 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.447937 1038 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.431640 987 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:17.470072 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.477083 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:17.478233 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 577441us
I20260806 20:16:17.480885 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:17.443032 1062 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:17.488660 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:17.491324 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1559) took 1302 ms. Trace:
I20260806 20:16:17.491765 30706 rpcz_store.cc:276] 0806 20:16:16.188616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.931850 (+743234us) service_pool.cc:227] Handling call
0806 20:16:16.932163 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:16.932443 (+ 280us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:16:17.311987 (+379544us) log.cc:842] Serialized 14 byte log entry
0806 20:16:17.490968 (+178981us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:17.491024 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:17.491075 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:17.491140 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:17.491263 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":240763,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"thread_start_us":538984,"threads_started":2}
I20260806 20:16:17.494906 30790 pending_rounds.cc:76] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Aborting all ops after (but not including) 5
I20260806 20:16:17.480651 944 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.499910 30790 pending_rounds.cc:97] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260806 20:16:17.502055 30792 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 9.
I20260806 20:16:17.473591 1038 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:17.509819 30790 raft_consensus.cc:2866] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:17.496559 1163 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.521523 30754 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:17.521855 30783 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:16:17.528793 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1939) took 1189 ms. Trace:
I20260806 20:16:17.534559 1040 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.536298 936 consensus_queue.cc:1046] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.008s
I20260806 20:16:17.529165 30783 rpcz_store.cc:276] 0806 20:16:16.339755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.913648 (+573893us) service_pool.cc:227] Handling call
0806 20:16:17.528705 (+615057us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:17.542343 935 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:17.543074 935 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.544075 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.544822 810 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.545410 810 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.546533 1025 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.546947 1066 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.547057 1025 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.547631 1066 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.548120 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.548815 30869 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:17.549830 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.505659 1019 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260806 20:16:17.537423 936 raft_consensus.cc:3032] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:17.499027 987 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.499598 1062 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.486055 1100 consensus_queue.cc:237] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:17.585595 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 57129.6us
I20260806 20:16:17.585893 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 357) took 1585 ms. Trace:
I20260806 20:16:17.563011 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.510144 944 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.547575 810 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:17.597119 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:17.586531 30713 rpcz_store.cc:276] 0806 20:16:16.004305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.274396 (+270091us) service_pool.cc:227] Handling call
0806 20:16:16.924873 (+650477us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:16:16.925142 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:16.977535 (+ 52393us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:16.977573 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:16.983763 (+ 6190us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:16.983810 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:17.585668 (+601858us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b540006bdd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:17.585729 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:17.585837 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78512640}
I20260806 20:16:17.588634 30756 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:17.588086 30867 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 8.
I20260806 20:16:17.595513 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:17.602258 30783 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 9.
I20260806 20:16:17.595940 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:17.555593 936 raft_consensus.cc:729] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:17.563175 1021 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.559176 1167 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:17.556115 1038 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.603515 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.625408 30703 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 8
I20260806 20:16:17.617142 30868 raft_consensus.cc:1226] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:17.622879 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.603529 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:17.624361 30673 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:17.619118 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.587105 935 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.622328 30788 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:17.618597 31389 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:17.602512 30757 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:17.629871 30865 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:17.623236 30676 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:17.632067 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 122199us
I20260806 20:16:17.640439 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:17.643435 30754 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:17.620571 1155 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.646826 30754 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:17.605437 1207 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:17.605150 1066 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.633436 1041 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.649544 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:17.602959 1200 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:17.651634 1206 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 20:16:17.652024 1198 log.cc:924] Time spent T 33e2a287c4de469eb5365fa7cdc6b409 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20260806 20:16:17.652438 944 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.684693 1200 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:17.652724 1202 log.cc:924] Time spent T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260806 20:16:17.608454 1040 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.646555 1021 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.654973 1204 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260806 20:16:17.650233 1155 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:17.656482 1210 log.cc:924] Time spent T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260806 20:16:17.657398 1197 log.cc:924] Time spent T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260806 20:16:17.657673 1126 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:17.658447 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1713) took 1981 ms. Trace:
I20260806 20:16:17.652619 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:17.659016 1012 consensus_queue.cc:1046] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:17.659866 1190 consensus_queue.cc:1046] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 20:16:17.663990 969 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.667493 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:17.655637 1196 log.cc:924] Time spent T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20260806 20:16:17.668118 1209 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:17.669780 1167 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.670128 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:17.638019 924 consensus_queue.cc:1035] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 2, Time since last communication: 0.161s
I20260806 20:16:17.634932 1076 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:17.645164 1009 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:17.671697 1207 raft_consensus.cc:686] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:17.672041 1038 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.672726 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.672968 987 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.651947 936 consensus_queue.cc:260] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.673527 30713 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:16:17.691326 30784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:17.684758 838 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 20:16:17.874723 865 raft_consensus.cc:3032] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:16:17.875502 865 raft_consensus.cc:729] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:17.755043 878 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.698787 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.876535 865 consensus_queue.cc:260] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.878129 865 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Advancing to term 4
W20260806 20:16:17.768011 961 consensus_meta.cc:287] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.571s user 0.007s sys 0.004s
I20260806 20:16:17.878825 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.878988 961 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.879581 30867 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:17.879105 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.880923 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.881667 30709 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:17.882421 1009 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:17.882716 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:17.883266 1155 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:17.883881 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.696925 1200 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:17.685133 1206 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:17.884913 1206 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:17.885231 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:17.886394 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 184206us
I20260806 20:16:17.724056 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.699211 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:17.851141 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 249711us
W20260806 20:16:17.820302 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 179916us
I20260806 20:16:17.862464 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.888545 31389 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:17.880901 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.696774 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:17.893363 30870 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:17.894222 30869 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:17.721954 30788 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:17.721458 30703 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:17.881346 1040 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:17.887548 30784 raft_consensus.cc:595] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:16:17.701666 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.903134 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:17.901767 30754 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:17.884614 1009 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:17.907795 30754 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:17.900548 30707 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:17.673610 1060 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:17.909946 30866 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:17.909587 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.929340 30703 raft_consensus.cc:2396] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260806 20:16:17.910578 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.876616 878 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:17.912650 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:17.913318 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 361) took 1868 ms. Trace:
I20260806 20:16:17.930070 30710 rpcz_store.cc:276] 0806 20:16:16.045129 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.902143 (+857014us) service_pool.cc:227] Handling call
0806 20:16:16.932994 (+ 30851us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:17.141213 (+208219us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:17.141249 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:17.141300 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:17.141353 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:17.913052 (+771699us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400174654. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:17.913103 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:17.913263 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":339813248}
I20260806 20:16:17.931066 30792 raft_consensus.cc:1261] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 20:16:17.897650 1076 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:17.915601 1021 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.699219 1120 raft_consensus.cc:3032] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 20:16:17.916546 30785 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1483) took 2703 ms (client timeout 1602 ms). Trace:
I20260806 20:16:17.917438 1165 raft_consensus.cc:3032] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:17.874775 838 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:17.918537 1189 raft_consensus.cc:3032] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:17.925951 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:17.915657 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:18.328887 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 467074us
I20260806 20:16:17.700091 1062 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:17.927018 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:17.897037 999 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:17.699167 1051 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:17.885288 1012 raft_consensus.cc:3032] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:17.936722 30865 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
W20260806 20:16:17.946092 30785 rpcz_store.cc:269] 0806 20:16:15.212958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:15.406902 (+193944us) service_pool.cc:227] Handling call
0806 20:16:17.916314 (+2509412us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54000573d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:17.916444 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3680027008}
I20260806 20:16:17.953758 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.319109 1226 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:17.955042 30867 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
W20260806 20:16:18.320225 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 382180us
I20260806 20:16:17.993039 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.146380 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:17.947427 1076 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:18.214794 1167 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:17.911513 1038 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:17.947135 1120 raft_consensus.cc:729] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:16:18.034659 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 47031.7us
I20260806 20:16:17.952687 1165 raft_consensus.cc:729] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:18.215688 1207 consensus_queue.cc:237] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:17.985517 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.022745 1208 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 20:16:18.273327 31420 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:18.213826 1126 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:17.932485 969 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:18.401794 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.212283 1206 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.021773 31421 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 431) took 1079 ms. Trace:
I20260806 20:16:17.924355 961 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:18.128710 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.024137 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:17.951974 30676 connection.cc:232] RPC call timeout handler was delayed by -1.0830328919983003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
W20260806 20:16:18.408421 30676 connection.cc:232] RPC call timeout handler was delayed by -1.0659328920010012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
I20260806 20:16:18.063602 30706 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 9.
I20260806 20:16:18.144675 30870 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
I20260806 20:16:18.035102 31416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48918 (request call id 430) took 1097 ms. Trace:
I20260806 20:16:17.926846 1200 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:17.955133 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.412205 30786 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:18.324458 936 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:17.954864 1060 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:18.208138 1231 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.225s user 0.001s sys 0.001s
I20260806 20:16:18.343070 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:17.885856 30705 rpcz_store.cc:276] 0806 20:16:15.677073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:15.687224 (+ 10151us) service_pool.cc:227] Handling call
0806 20:16:15.719452 (+ 32228us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:15.777535 (+ 58083us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:15.777564 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:15.778407 (+ 843us) log.cc:842] Serialized 26 byte log entry
0806 20:16:15.994297 (+215890us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:15.994347 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:15.994425 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:17.654149 (+1659724us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b24001a2268. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e5abcf4 00007f714e5a9dfb 00007f714e638559 00007f714e6381e5 00007f714e637740 00007f715131e578 00007f715131ecd8 00007f71512f66e8 00007f71512f1aea 00007f71512f0f36
0806 20:16:17.658048 (+ 3899us) raft_consensus.cc:1671] finished
0806 20:16:17.658073 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:17.658289 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56014,"raft.queue_time_us":59116,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":292,"spinlock_wait_cycles":78701440,"thread_start_us":215393,"threads_started":1,"wal-append.queue_time_us":562865}
I20260806 20:16:17.905676 924 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:18.339855 865 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:18.369925 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.351191 838 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:18.426870 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.427202 30789 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:18.377945 31389 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.351426 1189 raft_consensus.cc:729] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:18.426671 1165 consensus_queue.cc:260] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.426910 1200 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.440056 1165 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:18.440788 1044 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:18.440418 1189 consensus_queue.cc:260] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.442770 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.381600 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.390167 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.395510 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.396468 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.444098 30707 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:18.444234 30788 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:18.361776 1012 raft_consensus.cc:729] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:18.442436 1169 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:18.421825 1062 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:18.446503 1007 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.403156 30709 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 8.
I20260806 20:16:18.437568 865 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.447069 30713 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:18.447018 1062 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.448254 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:18.448859 30713 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 8
I20260806 20:16:18.449213 1062 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.450301 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.451557 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.448765 865 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.452637 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.402578 30835 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.446563 1012 consensus_queue.cc:260] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.454481 1167 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 20:16:18.454885 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 64018.9us
I20260806 20:16:18.456617 30835 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.407703 31421 rpcz_store.cc:276] 0806 20:16:16.942049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.942196 (+ 147us) service_pool.cc:227] Handling call
0806 20:16:18.021510 (+1079314us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54002cecd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:18.021668 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":364802432}
W20260806 20:16:18.408761 30676 connection.cc:232] RPC call timeout handler was delayed by -1.0659328920010012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
I20260806 20:16:18.458949 30676 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:18.459286 1167 raft_consensus.cc:2726] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:18.405252 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.409786 31416 rpcz_store.cc:276] 0806 20:16:16.938047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.938255 (+ 208us) service_pool.cc:227] Handling call
0806 20:16:18.034858 (+1096603us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b540058f2d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:18.035034 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8786688}
I20260806 20:16:18.415697 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.486960 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.487645 30870 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:18.488492 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.489423 1120 consensus_queue.cc:260] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.421265 30784 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:18.422171 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.406313 1155 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:18.424376 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.425494 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:17.953182 1209 raft_consensus.cc:686] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:18.495803 30706 raft_consensus.cc:2454] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 2.
I20260806 20:16:18.495705 30834 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:18.495630 30703 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:18.431787 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.425040 30792 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
W20260806 20:16:18.439144 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 61309us
I20260806 20:16:18.505364 30834 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:18.445221 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.446224 1169 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.540582 30705 raft_consensus.cc:2396] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 20:16:18.541929 1169 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.542764 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1726) took 1074 ms. Trace:
I20260806 20:16:18.543159 30705 rpcz_store.cc:276] 0806 20:16:17.468007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.541359 (+1073352us) service_pool.cc:227] Handling call
0806 20:16:18.542082 (+ 723us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.542435 (+ 353us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 20:16:18.542464 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.542504 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:16:18.542542 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.542599 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.542707 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:18.546728 30834 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:18.547132 30786 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:18.548133 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:18.445009 30867 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:18.404505 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.460636 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.319384 878 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:18.443068 1019 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260806 20:16:18.462371 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.442416 1060 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.465873 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:18.466485 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.469915 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.374861 1226 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.554958 961 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:18.561838 961 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:18.562229 961 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:18.562794 1226 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.470698 865 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:16:18.555172 1124 raft_consensus.cc:2726] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:18.441293 1200 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:18.471230 1190 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 20:16:18.478359 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1942) took 1893 ms. Trace:
I20260806 20:16:18.580693 30791 rpcz_store.cc:276] 0806 20:16:16.584752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:17.019840 (+435088us) service_pool.cc:227] Handling call
0806 20:16:17.042192 (+ 22352us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:17.183777 (+141585us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:17.183812 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:17.196393 (+ 12581us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:17.196446 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.478040 (+1281594us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54003d54d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:18.478102 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.478275 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":263422336}
I20260806 20:16:18.581568 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.484628 30835 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.582360 30869 raft_consensus.cc:2379] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
W20260806 20:16:18.582865 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 98393.6us
I20260806 20:16:18.418540 31388 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:18.489775 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.453354 30871 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:18.500599 1021 raft_consensus.cc:2726] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:18.588702 1051 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:18.500890 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.501616 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1725) took 1049 ms. Trace:
I20260806 20:16:18.501983 1155 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.585470 878 raft_consensus.cc:2781] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:18.591394 878 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:18.591777 878 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:18.590355 30703 rpcz_store.cc:276] 0806 20:16:17.451988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.500765 (+1048777us) service_pool.cc:227] Handling call
0806 20:16:18.501078 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.501277 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.501300 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.501348 (+ 48us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:18.501379 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.501425 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.501545 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:18.426105 924 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:18.503355 30783 raft_consensus.cc:2379] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 6: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:18.504156 1209 consensus_queue.cc:237] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.507488 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.512041 1120 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:18.517931 31417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 628) took 2492 ms. Trace:
I20260806 20:16:18.519526 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.521524 942 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.528980 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.617967 30863 raft_consensus.cc:2454] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 9.
I20260806 20:16:18.530148 1156 raft_consensus.cc:3032] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:18.534652 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.617236 30866 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:18.619640 30861 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:18.608100 924 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:18.505586 30708 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
W20260806 20:16:18.638700 30708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1563) took 1466 ms (client timeout 1574 ms). Trace:
W20260806 20:16:18.639132 30708 rpcz_store.cc:269] 0806 20:16:17.172623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.411360 (+1238737us) service_pool.cc:227] Handling call
0806 20:16:18.638616 (+227256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:18.662367 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1731) took 1148 ms. Trace:
I20260806 20:16:18.662964 30707 rpcz_store.cc:276] 0806 20:16:17.518815 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.560586 (+1041771us) service_pool.cc:227] Handling call
0806 20:16:18.560873 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.561042 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.561065 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.561102 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:18.561129 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.561176 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.662300 (+101124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:18.612982 1226 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 20:16:18.560148 1126 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:18.560827 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:18.676283 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 215710us
I20260806 20:16:18.575904 30713 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:18.576565 30834 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:16:18.453568 1009 consensus_meta.cc:287] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.511s user 0.011s sys 0.000s
W20260806 20:16:18.590687 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 172229us
I20260806 20:16:18.594013 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1732) took 1073 ms. Trace:
I20260806 20:16:18.584142 30835 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.679703 1009 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:18.680445 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 96391.7us
I20260806 20:16:18.681269 30835 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.681900 1009 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.682289 31423 raft_consensus.cc:1226] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 7. Ops: []
W20260806 20:16:18.682893 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 106428us
I20260806 20:16:18.682833 30703 rpcz_store.cc:276] 0806 20:16:17.521004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.593175 (+1072171us) service_pool.cc:227] Handling call
0806 20:16:18.593515 (+ 340us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.593709 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.593735 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.593777 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:18.593813 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.593862 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.593959 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:18.683979 987 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:18.684633 987 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:18.685241 987 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:18.677076 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.611357 31417 rpcz_store.cc:276] 0806 20:16:16.025039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.026454 (+ 1415us) service_pool.cc:227] Handling call
0806 20:16:16.026866 (+ 412us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:17.068058 (+1041192us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.517627 (+1449569us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400602a54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:18.517843 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2392620928}
I20260806 20:16:18.612254 30864 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:18.625250 1156 raft_consensus.cc:729] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:18.666049 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.738581 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.739274 30792 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:18.669554 1155 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:18.678747 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.740940 30708 raft_consensus.cc:2396] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 20:16:18.676498 30865 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:18.557911 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.681912 1012 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:18.746001 1195 raft_consensus.cc:3032] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:18.746487 1195 raft_consensus.cc:729] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:18.746758 987 raft_consensus.cc:686] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:18.747362 1195 consensus_queue.cc:260] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.747601 924 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.747792 987 consensus_queue.cc:237] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.726516 838 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.749403 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1953) took 1131 ms. Trace:
I20260806 20:16:18.749811 924 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.673596 1226 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:16:18.727969 1236 log.cc:924] Time spent T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 20:16:18.733965 1206 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.738243 1156 consensus_queue.cc:260] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.665907 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.760322 30864 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260806 20:16:18.751839 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.749818 30789 rpcz_store.cc:276] 0806 20:16:17.618438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:17.618575 (+ 137us) service_pool.cc:227] Handling call
0806 20:16:17.626601 (+ 8026us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.442588 (+815987us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.749175 (+306587us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b5400260f54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:18.749347 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1832130304}
I20260806 20:16:18.756744 30865 raft_consensus.cc:1226] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:18.763880 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.764642 30789 raft_consensus.cc:2379] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
W20260806 20:16:18.765676 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 84518.9us
I20260806 20:16:18.750908 838 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 10 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.754520 1226 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:18.770045 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 373) took 1047 ms. Trace:
I20260806 20:16:18.770496 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.771581 30865 raft_consensus.cc:1261] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 20:16:18.772011 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1738) took 1050 ms. Trace:
I20260806 20:16:18.771624 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.770496 30709 rpcz_store.cc:276] 0806 20:16:17.722932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.760734 (+1037802us) service_pool.cc:227] Handling call
0806 20:16:18.769313 (+ 8579us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.769743 (+ 430us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.769767 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.769808 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:18.769842 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.769894 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.769986 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:18.772882 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.773737 30789 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 3.
I20260806 20:16:18.772625 30708 rpcz_store.cc:276] 0806 20:16:17.721910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.760090 (+1038180us) service_pool.cc:227] Handling call
0806 20:16:18.771505 (+ 11415us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.771706 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.771731 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.771769 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:18.771807 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.771855 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.771950 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:18.774930 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 364219us
I20260806 20:16:18.770653 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:18.778506 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:18.780021 30789 raft_consensus.cc:3037] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:18.778357 1126 raft_consensus.cc:3032] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:18.779032 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.780886 1126 raft_consensus.cc:729] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:18.781877 1126 consensus_queue.cc:260] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.782881 1126 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:18.776321 838 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:18.786077 838 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.781960 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:18.788036 30783 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:18.771667 878 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.793035 878 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:18.793917 1156 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:18.794822 30757 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:18.795588 1241 raft_consensus.cc:2726] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:18.796715 30757 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:18.797500 1241 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:18.799012 30757 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:18.799913 1241 raft_consensus.cc:2726] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:18.800374 31423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 637) took 1176 ms. Trace:
I20260806 20:16:18.801045 30757 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:18.788898 838 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.803071 30676 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.760838 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 372) took 1086 ms. Trace:
I20260806 20:16:18.806201 30864 raft_consensus.cc:1261] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 20:16:18.806196 30703 rpcz_store.cc:276] 0806 20:16:17.674233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.760051 (+1085818us) service_pool.cc:227] Handling call
0806 20:16:18.760357 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.760527 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.760550 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.760588 (+ 38us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:18.760628 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.760677 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.760783 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:18.808492 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.809165 30703 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:16:18.810075 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.768074 936 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:18.812052 30865 raft_consensus.cc:1226] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 3. Current term is 4. Ops: []
I20260806 20:16:18.812974 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 374) took 1089 ms. Trace:
I20260806 20:16:18.813342 30708 rpcz_store.cc:276] 0806 20:16:17.723306 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.775811 (+1052505us) service_pool.cc:227] Handling call
0806 20:16:18.812522 (+ 36711us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.812691 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.812719 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.812756 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:18.812784 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.812834 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.812922 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:18.800760 31423 rpcz_store.cc:276] 0806 20:16:17.623819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:17.624256 (+ 437us) service_pool.cc:227] Handling call
0806 20:16:18.035252 (+410996us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.682693 (+647441us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.799300 (+116607us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b540058f2d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:18.800157 (+ 857us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b540058f2d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:18.800284 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2416909440}
W20260806 20:16:18.819145 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 51464.8us
I20260806 20:16:18.821329 1120 raft_consensus.cc:2741] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 20:16:18.822038 986 consensus_queue.cc:1046] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 20:16:18.821055 31388 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.821564 31419 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:16:18.828166 986 raft_consensus.cc:3032] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:18.828588 986 raft_consensus.cc:729] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:18.829586 986 consensus_queue.cc:260] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.830852 986 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:18.834532 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:18.836361 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.837152 30792 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 6.
W20260806 20:16:18.839394 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 54179.2us
I20260806 20:16:18.815251 1124 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:18.839636 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:18.843912 31388 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.844151 30703 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:18.815837 1241 raft_consensus.cc:2726] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:18.842552 30835 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:18.847723 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.852514 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:18.853494 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 362) took 2808 ms. Trace:
I20260806 20:16:18.853892 30712 rpcz_store.cc:276] 0806 20:16:16.061874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:16.901133 (+839259us) service_pool.cc:227] Handling call
0806 20:16:16.912265 (+ 11132us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:17.154004 (+241739us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 20:16:17.377897 (+223893us) log.cc:842] Serialized 14 byte log entry
0806 20:16:17.605746 (+227849us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:17.605799 (+ 53us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:17.605851 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.645347 (+1039496us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b18003a5520. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71575e68b1 00007f71575f3661 00007f714e720f96 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9
0806 20:16:18.853235 (+207888us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54001b4254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:18.853284 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.853413 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22796,"prepare.queue_time_us":70273,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":93,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":6106,"spinlock_wait_cycles":407064576,"thread_start_us":437122,"threads_started":2,"wal-append.queue_time_us":262350,"wal-append.run_cpu_time_us":1275,"wal-append.run_wall_time_us":1009924}
I20260806 20:16:18.853808 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:18.856547 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 375) took 1132 ms. Trace:
I20260806 20:16:18.856657 1021 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:18.857173 1021 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:18.857580 1021 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:18.856938 30712 rpcz_store.cc:276] 0806 20:16:17.724015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.855748 (+1131733us) service_pool.cc:227] Handling call
0806 20:16:18.856072 (+ 324us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.856259 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.856281 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.856324 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:18.856352 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.856401 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.856498 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:18.861560 838 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:18.848433 30834 leader_election.cc:302] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:18.812116 936 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.842972 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:18.849495 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.865279 30792 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:18.866439 31389 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:18.866791 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 90822.2us
I20260806 20:16:18.864050 30712 pending_rounds.cc:76] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 4
I20260806 20:16:18.862048 838 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:18.874250 31417 raft_consensus.cc:2354] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6.
I20260806 20:16:18.867705 30712 pending_rounds.cc:97] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260806 20:16:18.875749 30712 raft_consensus.cc:2866] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:18.878698 30703 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:18.880139 30703 pending_rounds.cc:76] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 4
I20260806 20:16:18.880396 30835 leader_election.cc:302] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.886847 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1739) took 1164 ms. Trace:
I20260806 20:16:18.888123 30835 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:18.876559 838 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:18.880404 30703 pending_rounds.cc:97] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260806 20:16:18.889696 30703 raft_consensus.cc:2866] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:18.894394 30783 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 10.
I20260806 20:16:18.896137 31389 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:18.898823 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:18.899526 30870 raft_consensus.cc:3037] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:18.887303 30707 rpcz_store.cc:276] 0806 20:16:17.722279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.760107 (+1037828us) service_pool.cc:227] Handling call
0806 20:16:18.760426 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.764686 (+ 4260us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:16:18.808070 (+ 43384us) log.cc:842] Serialized 14 byte log entry
0806 20:16:18.886514 (+ 78444us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.886563 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:18.886609 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.886669 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:18.886789 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":43710,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"thread_start_us":96220,"threads_started":2}
I20260806 20:16:18.905743 31388 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:18.876374 878 raft_consensus.cc:2781] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:18.911636 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.912431 30869 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:18.916394 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.917135 30792 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:18.820052 1076 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.876881 924 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:18.914050 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.923821 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.927316 30865 raft_consensus.cc:2396] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260806 20:16:18.890100 1165 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:18.932065 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.908713 1155 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 20:16:18.935484 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.935884 30756 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:18.936985 30756 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:18.937510 30707 raft_consensus.cc:1226] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:18.864921 961 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:18.877264 1208 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260806 20:16:18.943634 30710 raft_consensus.cc:2379] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:18.944006 30756 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:18.944769 30673 leader_election.cc:398] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6.
I20260806 20:16:18.946481 30673 leader_election.cc:401] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 20:16:18.948495 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.925691 924 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:18.951126 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.929872 1169 raft_consensus.cc:2781] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:18.889111 1226 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.952284 30756 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:18.965422 1167 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:18.956764 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:18.957165 30870 raft_consensus.cc:2454] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 8.
I20260806 20:16:18.921906 1256 raft_consensus.cc:2726] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 20:16:18.960732 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:18.972565 1207 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:18.974045 1239 raft_consensus.cc:979] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: : Instructing follower 42cabca48db7403b8523a3382a046043 to start an election
I20260806 20:16:18.986337 1207 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:18.973595 1241 raft_consensus.cc:2781] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 20:16:18.986822 1207 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:18.987138 1169 raft_consensus.cc:686] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:18.988163 961 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:18.973650 1030 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 6
I20260806 20:16:18.990250 1239 raft_consensus.cc:1067] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Signalling peer 42cabca48db7403b8523a3382a046043 to start an election
I20260806 20:16:18.955832 1120 consensus_queue.cc:1046] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260806 20:16:18.994077 1021 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.996415 1021 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:18.951937 1155 raft_consensus.cc:686] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:18.997378 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:19.009163 30864 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 7
I20260806 20:16:18.998906 30676 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:18.959919 924 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:18.949007 1156 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:18.986912 1241 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:19.005051 1076 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:18.967185 1226 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 10 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:18.988230 1169 consensus_queue.cc:237] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:18.946336 999 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:18.943022 936 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:19.003381 30789 raft_consensus.cc:2454] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:19.003202 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1741) took 1280 ms. Trace:
I20260806 20:16:19.055887 30709 rpcz_store.cc:276] 0806 20:16:17.722907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.773944 (+1051037us) service_pool.cc:227] Handling call
0806 20:16:18.815066 (+ 41122us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.863508 (+ 48442us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.863537 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.863573 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:18.863603 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:19.002958 (+139355us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b54000c6f54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:19.003010 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:19.003142 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111026048}
I20260806 20:16:18.973063 838 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:19.008462 1044 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:19.016654 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:19.059798 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:19.008383 1155 consensus_queue.cc:237] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:19.083779 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:19.084635 31418 raft_consensus.cc:2354] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7.
I20260806 20:16:19.021988 1204 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:19.085696 1204 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:19.023828 30676 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:19.025848 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:19.027757 1025 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:19.035770 1156 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.031538 31419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:19.103165 31419 raft_consensus.cc:2379] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 10: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:19.032424 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:19.103814 31423 raft_consensus.cc:2379] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 8: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:19.042328 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 370) took 1409 ms. Trace:
I20260806 20:16:19.003317 1207 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:19.053298 30792 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:19.054143 30864 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:16:19.060355 30870 raft_consensus.cc:3037] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:19.061976 1044 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:19.063335 838 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:19.064093 30756 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:19.064790 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:19.065092 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:19.065724 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:19.066977 30757 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:19.071929 1224 log.cc:924] Time spent T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.626s user 0.011s sys 0.004s
I20260806 20:16:19.073359 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:19.075376 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:19.055075 1120 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:19.052986 1248 log.cc:924] Time spent T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260806 20:16:19.061985 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:19.088323 31417 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:19.041527 1241 raft_consensus.cc:3037] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:19.031204 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:19.040916 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:19.104589 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 80844.1us
I20260806 20:16:18.970110 1255 consensus_queue.cc:1035] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.164s
I20260806 20:16:19.104231 30713 rpcz_store.cc:276] 0806 20:16:17.633162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.677771 (+1044609us) service_pool.cc:227] Handling call
0806 20:16:18.678081 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.678307 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:16:18.806065 (+127758us) log.cc:842] Serialized 14 byte log entry
0806 20:16:19.041941 (+235876us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:19.041999 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:19.042047 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:19.042112 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:19.042257 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":129812,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":10777,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":3618,"thread_start_us":286036,"threads_started":2}
I20260806 20:16:19.056285 999 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.085698 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:19.251716 30867 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:19.252399 30865 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:19.252791 30869 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 4.
I20260806 20:16:19.153623 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1730) took 1647 ms. Trace:
I20260806 20:16:19.246976 30863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1456) took 1323 ms. Trace:
I20260806 20:16:19.106249 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:19.170149 31420 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 20:16:19.151877 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:19.150430 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:19.248540 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 360458us
I20260806 20:16:19.255388 1189 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:19.277799 30835 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:19.278857 1040 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:19.279240 1040 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:19.279604 1040 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:19.227268 1252 log.cc:924] Time spent T 33e2a287c4de469eb5365fa7cdc6b409 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.182s user 0.003s sys 0.000s
I20260806 20:16:19.170571 30789 raft_consensus.cc:1226] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:19.281268 1019 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:19.134164 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:19.259704 1062 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:19.247323 924 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:19.282393 30786 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 9.
I20260806 20:16:19.161741 30706 raft_consensus.cc:1226] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:19.055590 986 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260806 20:16:19.257550 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 190653us
I20260806 20:16:19.255152 30784 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 3.
I20260806 20:16:19.106720 30832 leader_election.cc:398] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7.
I20260806 20:16:19.287364 30832 leader_election.cc:401] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20260806 20:16:19.284763 30711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1729) took 1786 ms (client timeout 1693 ms). Trace:
W20260806 20:16:19.256502 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 192490us
I20260806 20:16:19.288034 31389 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:19.117091 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 313) took 1661 ms. Trace:
I20260806 20:16:19.166590 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:19.276749 30788 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:19.289487 30787 rpcz_store.cc:276] 0806 20:16:17.455936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:17.456134 (+ 198us) service_pool.cc:227] Handling call
0806 20:16:17.470223 (+ 14089us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:17.617391 (+147168us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:17.617421 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:17.621841 (+ 4420us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:17.627113 (+ 5272us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:19.116836 (+1489723us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54003b15d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:19.116893 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:19.117021 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311612160}
I20260806 20:16:19.161072 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41700 (request call id 369) took 1533 ms. Trace:
I20260806 20:16:19.275710 30707 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:19.290854 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1961) took 1185 ms. Trace:
I20260806 20:16:19.275749 30705 rpcz_store.cc:276] 0806 20:16:17.506558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.544064 (+1037506us) service_pool.cc:227] Handling call
0806 20:16:18.638621 (+ 94557us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:18.649487 (+ 10866us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.649520 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:18.649572 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:18.649638 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.649704 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:19.153379 (+503675us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x7b54000c6a50. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:19.153554 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216544000}
I20260806 20:16:19.275754 31414 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:19.285396 30792 raft_consensus.cc:595] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:19.248605 1030 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6.
I20260806 20:16:19.292446 31421 raft_consensus.cc:1226] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:19.292443 30792 raft_consensus.cc:603] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:16:19.295022 30713 raft_consensus.cc:1261] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 20:16:19.295821 30709 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Advancing to term 10
W20260806 20:16:19.287869 30711 rpcz_store.cc:269] 0806 20:16:17.498492 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:17.520659 (+ 22167us) service_pool.cc:227] Handling call
0806 20:16:19.150241 (+1629582us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b5800001898. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:16:19.284689 (+134448us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25778560}
I20260806 20:16:19.246694 1204 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:19.291078 30710 rpcz_store.cc:276] 0806 20:16:17.627801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:17.931224 (+303423us) service_pool.cc:227] Handling call
0806 20:16:19.160836 (+1229612us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b5400030cd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:19.161000 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1330035456}
I20260806 20:16:19.303169 30789 rpcz_store.cc:276] 0806 20:16:18.104227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:19.060632 (+956405us) service_pool.cc:227] Handling call
0806 20:16:19.060911 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:19.280752 (+219841us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:19.289883 (+ 9131us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b5400012cd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:19.290052 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":251788544}
I20260806 20:16:19.298486 30835 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:19.296805 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:19.299423 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:19.302971 30757 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:19.300140 31389 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:19.104188 1156 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:19.154691 1165 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.304476 30676 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:19.156628 1126 raft_consensus.cc:3032] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:19.116909 1200 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:19.306731 30790 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 317) took 1849 ms. Trace:
I20260806 20:16:19.316617 30865 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 10.
I20260806 20:16:19.322578 30785 raft_consensus.cc:1261] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 20:16:19.306640 30861 raft_consensus.cc:1261] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:16:19.305636 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:19.305718 30871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:16:19.299818 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 195775us
I20260806 20:16:19.292795 838 raft_consensus.cc:2781] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 20:16:19.338572 31388 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:19.348770 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:19.165673 1207 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:19.351428 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:19.294292 878 raft_consensus.cc:686] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:19.341604 30788 raft_consensus.cc:1226] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:19.275369 999 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:19.276420 30863 rpcz_store.cc:276] 0806 20:16:17.923244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.726156 (+802912us) service_pool.cc:227] Handling call
0806 20:16:18.726545 (+ 389us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:18.833981 (+107436us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.834010 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:18.834525 (+ 515us) log.cc:842] Serialized 26 byte log entry
0806 20:16:18.922605 (+ 88080us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:18.922645 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:19.081946 (+159301us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b540019d554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:19.082000 (+ 54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:19.246606 (+164606us) raft_consensus.cc:1671] finished
0806 20:16:19.246627 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:19.246880 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5557,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":18909,"spinlock_wait_cycles":245960960,"thread_start_us":87673,"threads_started":1,"wal-append.queue_time_us":257862}
I20260806 20:16:19.342245 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:19.355312 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1724) took 1911 ms. Trace:
W20260806 20:16:19.342594 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 44192.6us
I20260806 20:16:19.343693 30870 raft_consensus.cc:2454] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:19.356504 30706 rpcz_store.cc:276] 0806 20:16:17.443999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.502717 (+1058718us) service_pool.cc:227] Handling call
0806 20:16:19.161538 (+658821us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:19.292006 (+130468us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:19.355119 (+ 63113us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b5400030cd0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:19.355256 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1514345728}
I20260806 20:16:19.345091 30864 raft_consensus.cc:2379] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:19.349712 30713 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 7
I20260806 20:16:19.350347 30792 raft_consensus.cc:1226] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260806 20:16:19.340651 31389 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:16:19.353194 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 50309.5us
I20260806 20:16:19.354049 30711 raft_consensus.cc:2396] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260806 20:16:19.358700 30835 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:19.354683 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:19.355495 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:19.297284 30655 placement_policy_util.cc:244] tablet 5ba1969a3911423e8e51fc4f2962515d: not considering replicas for movement since the tablet's status is 'UNAVAILABLE'
I20260806 20:16:19.318150 1195 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:19.284773 1120 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:19.345819 30790 rpcz_store.cc:276] 0806 20:16:17.457409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:17.466435 (+ 9026us) service_pool.cc:227] Handling call
0806 20:16:17.485993 (+ 19558us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 20:16:17.609812 (+123819us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:17.609848 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 20:16:17.619386 (+ 9538us) log.cc:842] Serialized 52 byte log entry
0806 20:16:17.887442 (+268056us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:17.887488 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:17.887639 (+ 151us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:18.727223 (+839584us) spinlock_profiling.cc:243] Waited 4.42 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:19.071569 (+344346us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b2400105248. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e5abcf4 00007f714e5a9dfb 00007f714e638559 00007f714e6381e5 00007f714e63782d 00007f71512da46e 00007f71512f1c0c 00007f71512f0f36 00007f71512f0b58 00007f71512fa551
0806 20:16:19.306517 (+234948us) raft_consensus.cc:1671] finished
0806 20:16:19.306538 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:19.306669 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48899,"raft.queue_time_us":67440,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":12934,"spinlock_wait_cycles":64561280,"thread_start_us":242215,"threads_started":1,"wal-append.queue_time_us":783844}
I20260806 20:16:19.359333 30757 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:19.348698 1025 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:19.349689 838 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.345070 1256 raft_consensus.cc:2726] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519) with higher term. Message: Invalid argument: T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7.
I20260806 20:16:19.387367 30757 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:19.393639 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 53069.4us
I20260806 20:16:19.394384 31389 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:19.395139 1009 raft_consensus.cc:2781] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:19.395226 1120 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:19.395380 1208 raft_consensus.cc:2726] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:19.358415 30832 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:19.395800 1009 raft_consensus.cc:686] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:19.338294 1165 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:19.398010 1009 consensus_queue.cc:237] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:19.305966 969 consensus_queue.cc:1035] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20260806 20:16:19.340754 1126 raft_consensus.cc:729] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:19.348356 1222 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:19.401724 1222 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.402114 1222 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:19.402444 878 consensus_queue.cc:237] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:19.387068 1207 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:19.407981 1207 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:19.410142 1255 consensus_queue.cc:1046] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:16:19.348593 1190 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:19.411062 1190 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.410548 1222 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:19.411432 1190 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:19.394456 30757 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:19.397259 30832 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:19.414502 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1451) took 1492 ms. Trace:
I20260806 20:16:19.415148 969 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:19.393802 838 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:19.415710 969 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.416188 969 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:19.419051 30835 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:19.419626 31389 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:16:19.419553 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 78329.1us
I20260806 20:16:19.423964 30835 leader_election.cc:302] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:19.420825 1062 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:19.301774 1206 consensus_queue.cc:1046] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.090s
I20260806 20:16:19.395203 30705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6"
dest_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:59116
I20260806 20:16:19.390503 1167 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:19.435285 30835 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:19.424651 30713 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:19.394140 1025 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.434026 30705 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:16:19.422492 30676 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:19.414732 935 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:19.437618 30713 pending_rounds.cc:76] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 2
I20260806 20:16:19.444917 30713 pending_rounds.cc:97] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260806 20:16:19.390389 1239 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:19.414952 30868 rpcz_store.cc:276] 0806 20:16:17.921767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.632947 (+711180us) service_pool.cc:227] Handling call
0806 20:16:18.665287 (+ 32340us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:18.741884 (+ 76597us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:16:18.741919 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:18.764335 (+ 22416us) log.cc:842] Serialized 26 byte log entry
0806 20:16:18.905514 (+141179us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:18.905554 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:18.905618 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:19.414286 (+508668us) raft_consensus.cc:1671] finished
0806 20:16:19.414312 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:19.414444 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5021,"raft.queue_time_us":9613,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":4715,"thread_start_us":117278,"threads_started":1,"wal-append.queue_time_us":200534}
I20260806 20:16:19.437520 30834 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:19.430156 30707 raft_consensus.cc:2454] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 9.
I20260806 20:16:19.428386 1200 raft_consensus.cc:686] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:19.410156 1207 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:19.440713 30705 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:19.442216 30710 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:16:19.415827 30866 raft_consensus.cc:2932] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:19.467298 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:19.468041 30788 raft_consensus.cc:2362] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7. Re-sending same reply.
I20260806 20:16:19.443809 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:19.445259 30713 raft_consensus.cc:2866] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:19.414459 30832 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:19.448293 30757 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:19.451871 30707 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:19.438592 30787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "42cabca48db7403b8523a3382a046043"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:19.449651 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:19.452752 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:19.432739 1124 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 20:16:19.472608 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 58210.5us
I20260806 20:16:19.471987 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:19.473209 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:19.474076 30787 raft_consensus.cc:595] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Received request to transfer leadership to TS 42cabca48db7403b8523a3382a046043
I20260806 20:16:19.477232 30787 raft_consensus.cc:603] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:16:19.477249 30785 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:19.439272 1226 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:19.439092 1167 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:19.442407 1025 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:19.454753 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:19.435595 1256 raft_consensus.cc:2726] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:19.439640 1019 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:19.481989 30789 raft_consensus.cc:2354] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 9. Current term is 10.
I20260806 20:16:19.482304 30676 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:16:19.443078 30712 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.018s sys 0.010s
I20260806 20:16:19.438820 1076 raft_consensus.cc:2741] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 20:16:19.435242 1062 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:19.390244 1257 consensus_queue.cc:1046] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260806 20:16:19.472348 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:19.630133 30792 raft_consensus.cc:2379] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 9: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:16:19.492352 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 90517.3us
I20260806 20:16:19.452095 986 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:16:19.580077 30712 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s user 0.021s sys 0.018s
I20260806 20:16:19.486333 942 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:19.508983 30707 raft_consensus.cc:1261] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 20:16:19.454838 1206 raft_consensus.cc:3032] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:19.453240 1051 raft_consensus.cc:2726] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 20:16:19.619829 31422 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:19.548002 30790 raft_consensus.cc:1261] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260806 20:16:19.619830 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 200269us
I20260806 20:16:19.410173 1126 consensus_queue.cc:260] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:19.486809 1169 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:19.490820 1012 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:19.674510 31389 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:19.675915 30754 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:19.636708 987 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:19.456007 961 consensus_queue.cc:1035] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.164s
I20260806 20:16:19.708173 30835 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:19.508600 1167 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:19.484051 1222 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:19.636942 1124 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 20:16:19.730221 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 292769us
W20260806 20:16:19.732944 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 58518.7us
I20260806 20:16:19.449230 1258 consensus_queue.cc:1046] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.223s
W20260806 20:16:19.756057 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 279623us
W20260806 20:16:19.788842 30867 consensus_meta.cc:287] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.505s user 0.006s sys 0.001s
I20260806 20:16:19.800597 30867 raft_consensus.cc:2396] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
W20260806 20:16:19.790799 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 360928us
I20260806 20:16:19.638267 986 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.638693 924 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:19.638515 1206 raft_consensus.cc:729] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:19.710918 1169 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:19.735334 1012 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.824980 30710 raft_consensus.cc:1261] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 20:16:19.801798 30757 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:19.629570 1019 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:19.844502 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 136414us
I20260806 20:16:19.847785 30835 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:19.738775 1044 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:19.490001 1239 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.329s
I20260806 20:16:19.817315 1120 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:19.843531 1155 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:19.844993 30783 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:19.793526 1124 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:19.865178 30791 raft_consensus.cc:1261] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260806 20:16:19.865270 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 383007us
I20260806 20:16:19.817840 1126 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:19.811702 986 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:19.839051 30785 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
W20260806 20:16:19.855937 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 59886.6us
W20260806 20:16:19.863476 30709 consensus_meta.cc:287] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.564s user 0.008s sys 0.000s
I20260806 20:16:19.811144 1254 raft_consensus.cc:2726] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 20:16:19.831215 1169 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:19.638247 1209 consensus_queue.cc:1046] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.179s
I20260806 20:16:19.868050 30709 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 10.
I20260806 20:16:19.832194 1012 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:19.874136 30676 leader_election.cc:302] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
W20260806 20:16:19.859130 1040 consensus_meta.cc:287] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.575s user 0.006s sys 0.004s
I20260806 20:16:19.905422 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1577) took 1552 ms. Trace:
I20260806 20:16:19.925089 30703 rpcz_store.cc:276] 0806 20:16:18.352243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.879609 (+527366us) service_pool.cc:227] Handling call
0806 20:16:18.879972 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:18.909503 (+ 29531us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.909587 (+ 84us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:18.910526 (+ 939us) log.cc:842] Serialized 26 byte log entry
0806 20:16:19.255501 (+344975us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:19.255534 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:19.255602 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:19.882930 (+627328us) raft_consensus.cc:1671] finished
0806 20:16:19.903868 (+ 20938us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:19.904828 (+ 960us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":293556,"threads_started":1,"wal-append.queue_time_us":683692}
I20260806 20:16:19.816013 1200 consensus_queue.cc:237] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:19.864042 1195 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:19.869204 1124 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:19.954772 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 80750.7us
W20260806 20:16:19.913162 1189 consensus_meta.cc:287] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.635s user 0.005s sys 0.005s
W20260806 20:16:19.956260 1267 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.142s user 0.001s sys 0.001s
W20260806 20:16:19.968684 1269 log.cc:924] Time spent T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.110s user 0.002s sys 0.000s
I20260806 20:16:19.902045 1155 raft_consensus.cc:686] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:20.001987 1282 raft_consensus.cc:2781] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 20:16:20.011813 1282 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:20.012215 1282 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:20.004060 1155 consensus_queue.cc:237] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.004413 1062 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.015453 1169 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.016356 986 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.000299 30708 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:19.930146 1040 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.024418 1040 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:20.025075 1285 raft_consensus.cc:2726] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 20:16:19.970463 1261 log.cc:924] Time spent T 3e24eef6446040b8abbe977d4dd5a3ca P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
W20260806 20:16:19.932915 1266 log.cc:924] Time spent T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20260806 20:16:19.937819 1051 raft_consensus.cc:2781] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:20.026441 1051 raft_consensus.cc:686] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:20.027174 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1576) took 1680 ms. Trace:
W20260806 20:16:20.005160 1165 consensus_meta.cc:287] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.607s user 0.007s sys 0.002s
I20260806 20:16:20.027380 1051 consensus_queue.cc:237] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.027920 1165 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.028393 1025 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:20.023767 838 consensus_meta.cc:287] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.600s user 0.009s sys 0.000s
W20260806 20:16:19.995431 1263 log.cc:924] Time spent T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.174s user 0.001s sys 0.001s
I20260806 20:16:19.946290 1279 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:20.031150 1190 consensus_meta.cc:287] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.619s user 0.008s sys 0.001s
I20260806 20:16:19.930670 29419 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 20:16:19.943657 1274 raft_consensus.cc:2726] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:16:20.030831 1279 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:19.943105 1275 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:19.975530 1286 consensus_queue.cc:1035] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
W20260806 20:16:20.042490 1241 consensus_meta.cc:287] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.741s user 0.009s sys 0.000s
I20260806 20:16:20.033800 30712 rpcz_store.cc:276] 0806 20:16:18.347115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.863226 (+516111us) service_pool.cc:227] Handling call
0806 20:16:18.863882 (+ 656us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:18.888192 (+ 24310us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:18.888251 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:18.888757 (+ 506us) log.cc:842] Serialized 26 byte log entry
0806 20:16:19.632699 (+743942us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:19.632738 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:19.632804 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:20.026959 (+394155us) raft_consensus.cc:1671] finished
0806 20:16:20.026979 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:20.027111 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":706639,"threads_started":1,"wal-append.queue_time_us":953360}
W20260806 20:16:20.044775 999 consensus_meta.cc:287] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.620s user 0.005s sys 0.005s
W20260806 20:16:20.036111 31417 consensus_meta.cc:287] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.752s user 0.006s sys 0.002s
I20260806 20:16:20.051534 31417 raft_consensus.cc:2396] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 20:16:20.052044 31417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49046 (request call id 642) took 1042 ms. Trace:
I20260806 20:16:20.052398 31417 rpcz_store.cc:276] 0806 20:16:19.010110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:19.010243 (+ 133us) service_pool.cc:227] Handling call
0806 20:16:20.051983 (+1041740us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:20.021080 1156 consensus_meta.cc:287] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.669s user 0.000s sys 0.008s
W20260806 20:16:20.038170 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 170908us
I20260806 20:16:19.999688 987 raft_consensus.cc:686] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:16:20.041674 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 46587.2us
I20260806 20:16:20.049155 1169 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.058396 1204 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:20.060364 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:20.055955 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1504) took 1308 ms. Trace:
I20260806 20:16:20.032886 986 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:19.927942 1120 raft_consensus.cc:686] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:20.038915 30708 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:16:19.989708 1195 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:20.033776 1167 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.030089 1165 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:19.923990 30601 catalog_manager.cc:5495] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 7, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 7 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
W20260806 20:16:20.069288 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 267578us
I20260806 20:16:20.061951 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:20.070061 30703 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:20.060396 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:20.071026 30785 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
W20260806 20:16:20.042016 30705 consensus_meta.cc:287] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.586s user 0.008s sys 0.000s
W20260806 20:16:20.046136 969 consensus_meta.cc:287] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.629s user 0.007s sys 0.002s
I20260806 20:16:20.055207 961 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:20.076488 30789 raft_consensus.cc:1226] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:20.076498 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:20.078649 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:20.078890 30788 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:20.054700 1041 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:20.072530 30705 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.044063 1190 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.089290 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:20.064822 1277 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:20.085137 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 237427us
I20260806 20:16:20.085356 30868 raft_consensus.cc:1226] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 20:16:20.065969 1025 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:20.088311 31415 raft_consensus.cc:1226] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:20.099152 30868 raft_consensus.cc:1226] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 5. Current term is 7. Ops: []
W20260806 20:16:20.080736 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 293503us
I20260806 20:16:20.063634 1239 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:20.065084 1030 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:20.100274 1239 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.094255 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:20.094486 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:20.056962 999 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.072178 1195 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:20.100200 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:20.054572 1241 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.055877 838 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.091645 30705 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.077138 30756 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:20.059604 1156 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.105965 30706 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260806 20:16:20.095238 1189 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:20.051949 1275 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.108100 30711 raft_consensus.cc:1261] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 20:16:20.109761 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:20.112066 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:20.119673 30835 leader_election.cc:398] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 election: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 9. Current term is 10.
I20260806 20:16:20.123942 30835 leader_election.cc:401] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
W20260806 20:16:20.133162 30866 consensus_meta.cc:287] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.653s user 0.007s sys 0.006s
I20260806 20:16:20.133929 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:59116 (request call id 1592) took 1105 ms. Trace:
W20260806 20:16:20.107760 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 475688us
I20260806 20:16:20.134351 30705 rpcz_store.cc:276] 0806 20:16:19.032959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:19.386627 (+353668us) service_pool.cc:227] Handling call
0806 20:16:20.117197 (+730570us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:20.101478 1019 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:20.140156 30676 connection.cc:232] RPC call timeout handler was delayed by -1.0230803320009727s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
I20260806 20:16:20.078706 969 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.144181 31388 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:20.148882 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 78115.5us
I20260806 20:16:20.150549 30757 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:20.154484 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:20.155745 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 235) took 1250 ms. Trace:
I20260806 20:16:20.157150 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:20.158102 30792 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:20.156132 30866 rpcz_store.cc:276] 0806 20:16:18.905800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.906000 (+ 200us) service_pool.cc:227] Handling call
0806 20:16:18.926108 (+ 20108us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 20:16:18.978888 (+ 52780us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:16:18.979349 (+ 461us) log.cc:842] Serialized 14 byte log entry
0806 20:16:19.220884 (+241535us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 20:16:19.287233 (+ 66349us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b2c001497e8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:16:19.415543 (+128310us) log.cc:842] Serialized 624 byte log entry
0806 20:16:19.415695 (+ 152us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:20.133895 (+718200us) log.cc:842] Serialized 14 byte log entry
0806 20:16:20.155321 (+ 21426us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:20.155400 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:20.155528 (+ 128us) raft_consensus.cc:1671] finished
0806 20:16:20.155542 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:20.155688 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23242,"prepare.queue_time_us":272,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":20678,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":8839,"spinlock_wait_cycles":60883328,"thread_start_us":256859,"threads_started":2,"wal-append.queue_time_us":503823}
I20260806 20:16:20.162123 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:20.102161 1277 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.160367 1044 raft_consensus.cc:2781] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:16:20.101238 961 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.165122 30676 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:20.165149 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 44154.9us
I20260806 20:16:20.167250 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:20.171571 30868 raft_consensus.cc:1226] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:20.173689 30785 raft_consensus.cc:1226] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 8. Ops: []
I20260806 20:16:20.173907 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:20.178154 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:20.179738 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:20.180368 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:20.180871 30870 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:20.184628 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:20.105549 1030 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.135758 1189 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.147832 1258 consensus_queue.cc:1035] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260806 20:16:20.142405 987 consensus_queue.cc:237] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.188563 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:20.378347 1241 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:20.224253 30865 raft_consensus.cc:1226] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:20.377686 1280 raft_consensus.cc:482] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:20.382427 1280 raft_consensus.cc:504] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.384546 1280 leader_election.cc:288] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:20.385833 1275 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:20.386926 1239 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.257181 30705 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 20:16:20.378063 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 229715us
I20260806 20:16:20.200552 30870 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:20.398998 30861 raft_consensus.cc:1226] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 4. Current term is 5. Ops: []
W20260806 20:16:20.402470 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 252008us
I20260806 20:16:20.219728 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:20.189435 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:20.249130 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 129574us
I20260806 20:16:20.409703 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:20.410358 30835 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:20.410471 30707 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:20.218855 30863 raft_consensus.cc:1226] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:20.218164 31417 raft_consensus.cc:1226] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: []
I20260806 20:16:20.221170 30866 raft_consensus.cc:2454] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
W20260806 20:16:20.206532 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 129481us
I20260806 20:16:20.420797 30834 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:20.415333 1239 consensus_queue.cc:1046] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.008s
I20260806 20:16:20.385962 1190 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.424379 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:20.424609 30703 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:20.346562 1290 log.cc:924] Time spent T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
W20260806 20:16:20.345535 1268 log.cc:924] Time spent T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20260806 20:16:20.438844 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:20.274962 942 consensus_meta.cc:287] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.600s user 0.009s sys 0.000s
I20260806 20:16:20.219846 1278 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:20.175416 1120 consensus_queue.cc:237] T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.447296 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:20.450429 1155 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:20.464525 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:20.441411 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 297909us
I20260806 20:16:20.478420 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:20.371084 1167 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:20.471066 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 294057us
I20260806 20:16:20.216658 986 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:20.229950 1255 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 9. Current term is 10.
I20260806 20:16:20.432961 1040 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:20.478322 30705 raft_consensus.cc:1226] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 20:16:20.375814 1156 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.426770 1239 raft_consensus.cc:3032] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:20.482640 31421 raft_consensus.cc:1226] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:20.274660 1307 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:20.486428 924 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:16:20.487182 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:20.473227 1278 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:16:20.473109 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 58566.6us
I20260806 20:16:20.572057 30676 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:20.437485 838 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.346881 1279 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.426218 1189 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.429383 999 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.438930 1277 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:20.503671 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:20.438503 1241 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:20.274398 1294 consensus_queue.cc:1046] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.295s
I20260806 20:16:20.533247 30703 raft_consensus.cc:2454] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:20.563597 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:20.332636 1274 consensus_queue.cc:1035] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.150s
I20260806 20:16:20.488147 31389 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:20.228509 1012 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.562073 30792 raft_consensus.cc:2454] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:20.503531 986 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:20.478557 942 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:20.478865 1044 raft_consensus.cc:686] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:20.475528 1295 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:20.504438 1155 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.565009 1239 raft_consensus.cc:729] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:21.008800 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:21.009202 30864 raft_consensus.cc:1261] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 20:16:21.009514 1239 consensus_queue.cc:260] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:21.010722 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 600401us
I20260806 20:16:20.556425 1293 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:20.879709 29419 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 20:16:21.067764 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 579640us
I20260806 20:16:21.071771 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:21.073056 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:21.073984 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 62365.7us
W20260806 20:16:21.075368 30787 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.005s sys 0.006s
W20260806 20:16:21.076644 30787 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.006s sys 0.007s
W20260806 20:16:21.075757 1051 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.888s user 0.000s sys 0.012s
I20260806 20:16:21.077461 30710 raft_consensus.cc:1226] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:21.077883 1051 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s user 0.000s sys 0.013s
I20260806 20:16:21.079806 30710 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 20:16:21.083719 30790 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:21.084318 31389 leader_election.cc:302] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:20.901353 30601 catalog_manager.cc:5495] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 3 to 6, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 6 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:21.093684 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:20.569573 1307 raft_consensus.cc:482] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:16:20.870290 1304 log.cc:924] Time spent T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.217s user 0.000s sys 0.000s
I20260806 20:16:21.098658 30705 raft_consensus.cc:2396] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 20:16:20.605764 1278 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
W20260806 20:16:20.691435 1260 log.cc:924] Time spent T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20260806 20:16:21.101949 30707 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 9: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
W20260806 20:16:21.102799 30599 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41550 (request call id 238) took 14267 ms (client timeout 15000 ms). Trace:
W20260806 20:16:21.111263 30599 rpcz_store.cc:269] 0806 20:16:06.835571 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:06.853695 (+ 18124us) service_pool.cc:227] Handling call
0806 20:16:21.102580 (+14248885us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:16:15.500461 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:16:15.501256 (+ 795us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:16:15.501320 (+ 64us) write_op.cc:435] Acquired schema lock
0806 20:16:15.501369 (+ 49us) tablet.cc:660] Decoding operations
0806 20:16:15.530309 (+ 28940us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:16:15.530371 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0806 20:16:15.530397 (+ 26us) tablet.cc:683] Acquiring locks for 13 operations
0806 20:16:15.531442 (+ 1045us) tablet.cc:699] Row locks acquired
0806 20:16:15.531462 (+ 20us) write_op.cc:260] PREPARE: finished
0806 20:16:15.531613 (+ 151us) write_op.cc:270] Start()
0806 20:16:15.531729 (+ 116us) write_op.cc:276] Timestamp: P: 1786047375531561 usec, L: 0
0806 20:16:15.531749 (+ 20us) op_driver.cc:331] REPLICATION: starting
0806 20:16:15.532454 (+ 705us) log.cc:842] Serialized 5232 byte log entry
0806 20:16:17.447045 (+1914591us) op_driver.cc:444] REPLICATION: finished
0806 20:16:17.650882 (+203837us) write_op.cc:301] APPLY: starting
0806 20:16:17.652197 (+ 1315us) tablet.cc:1365] starting BulkCheckPresence
0806 20:16:17.652523 (+ 326us) tablet.cc:1368] finished BulkCheckPresence
0806 20:16:17.652651 (+ 128us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:16:17.654586 (+ 1935us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:16:17.654965 (+ 379us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0806 20:16:17.655007 (+ 42us) write_op.cc:312] APPLY: finished
0806 20:16:17.655523 (+ 516us) log.cc:842] Serialized 95 byte log entry
0806 20:16:17.658629 (+ 3106us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:17.658884 (+ 255us) write_op.cc:454] Released schema lock
0806 20:16:17.658952 (+ 68us) write_op.cc:341] FINISH: Updating metrics
0806 20:16:17.659339 (+ 387us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:17.659365 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2664361,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":181866,"apply.run_cpu_time_us":5014,"apply.run_wall_time_us":16305,"num_ops":13,"prepare.queue_time_us":463090,"prepare.run_cpu_time_us":37481,"prepare.run_wall_time_us":603361,"raft.queue_time_us":521403,"raft.run_cpu_time_us":32746,"raft.run_wall_time_us":64836,"replication_time_us":1915144,"thread_start_us":1181256,"threads_started":4,"wal-append.queue_time_us":835696,"wal-append.run_cpu_time_us":48287,"wal-append.run_wall_time_us":2288106}]]}
I20260806 20:16:20.567202 1255 raft_consensus.cc:482] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:21.116690 1255 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.119562 1255 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:20.876199 1262 raft_consensus.cc:3032] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:21.120277 1262 raft_consensus.cc:729] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:21.121294 1262 consensus_queue.cc:260] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.835245 961 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:20.837062 1275 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.100867 1307 raft_consensus.cc:3037] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:21.135460 1044 consensus_queue.cc:237] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.609871 1258 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:21.059717 1295 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:21.137528 936 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:21.137715 1012 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:21.138932 1051 raft_consensus.cc:482] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.139454 1313 consensus_queue.cc:1035] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:16:21.139588 1226 consensus_queue.cc:1046] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 20:16:21.140400 1259 consensus_queue.cc:1046] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:16:21.140879 1279 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:21.141623 1155 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:21.143492 1255 raft_consensus.cc:482] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.070439 986 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:21.135485 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 563500us
I20260806 20:16:21.259770 30676 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:21.084265 1241 raft_consensus.cc:504] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:20.765275 1156 consensus_queue.cc:1035] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.388s
I20260806 20:16:21.154671 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1981) took 1707 ms. Trace:
I20260806 20:16:21.154768 30601 catalog_manager.cc:5495] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 4 to 5, leader changed from 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 5 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:21.099102 924 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:21.106954 1303 raft_consensus.cc:2781] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:21.094832 1041 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 20:16:21.256614 1124 consensus_meta.cc:287] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.220s user 0.007s sys 0.004s
I20260806 20:16:21.169653 1278 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:20.566195 1040 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.172365 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:21.174299 30756 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:21.134629 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 20:16:21.176932 30757 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.000s sys 0.023s
I20260806 20:16:21.094595 1209 raft_consensus.cc:482] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:16:21.185536 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 109875us
W20260806 20:16:21.188858 1311 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.024s sys 0.011s
I20260806 20:16:21.140286 1189 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.099498 942 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:21.146996 1270 consensus_queue.cc:1035] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
W20260806 20:16:21.215013 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 184054us
I20260806 20:16:21.183019 1293 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:21.254146 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:21.288264 30713 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:21.265038 30790 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
I20260806 20:16:21.198292 1295 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:21.167544 1195 consensus_meta.cc:287] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.043s user 0.006s sys 0.003s
I20260806 20:16:21.268869 30787 rpcz_store.cc:276] 0806 20:16:19.447668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:19.508350 (+ 60682us) service_pool.cc:227] Handling call
0806 20:16:19.673470 (+165120us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:19.709998 (+ 36528us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:19.710028 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:20.283275 (+573247us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b2c000002e8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:16:20.332156 (+ 48881us) log.cc:842] Serialized 26 byte log entry
0806 20:16:21.079287 (+747131us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:16:21.079328 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:21.079403 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:21.154357 (+ 74954us) raft_consensus.cc:1671] finished
0806 20:16:21.154378 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:21.154600 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":170757,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":6603,"spinlock_wait_cycles":78258560,"thread_start_us":645546,"threads_started":1,"wal-append.queue_time_us":750803}
W20260806 20:16:21.288154 30757 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.846s user 0.000s sys 0.024s
W20260806 20:16:21.331236 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 913923us
I20260806 20:16:21.332921 30757 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:21.271132 30676 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:21.333961 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 226363us
W20260806 20:16:21.334395 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 63367.7us
I20260806 20:16:21.336035 30676 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:21.254906 936 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.182610 1275 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:21.286713 30755 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.011s sys 0.011s
W20260806 20:16:21.307749 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 133537us
I20260806 20:16:21.191154 1301 consensus_queue.cc:1046] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260806 20:16:21.301859 1280 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:21.270750 1012 raft_consensus.cc:3037] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:21.305749 942 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.305842 1124 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:21.302217 1311 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.878s user 0.024s sys 0.013s
I20260806 20:16:21.270424 1156 consensus_queue.cc:1225] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer 58d59beb597f4ed99d84d102ddb41ff5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 20:16:21.232549 1312 consensus_queue.cc:1046] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20260806 20:16:21.323704 30601 catalog_manager.cc:5495] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 6 to 7, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 7 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:21.305512 1209 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.324949 1331 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 20:16:21.321813 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:21.326785 1278 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:21.232815 1051 raft_consensus.cc:504] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.305574 1189 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.258230 1255 raft_consensus.cc:504] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.284307 1241 leader_election.cc:288] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.274608 969 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:21.343763 30713 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:21.382388 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:21.351289 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 180794us
W20260806 20:16:21.351589 30755 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.879s user 0.012s sys 0.012s
I20260806 20:16:21.322790 1303 raft_consensus.cc:686] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
W20260806 20:16:21.268628 1298 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.024s sys 0.012s
I20260806 20:16:21.199378 1258 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.387969 30867 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:21.359689 30757 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:21.366545 1041 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.378202 1156 consensus_queue.cc:1046] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:16:21.386948 31389 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:21.302160 1279 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.306427 1195 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.353755 1226 consensus_queue.cc:1035] T f92e9dc9db864f0b8e238a39be31b12e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260806 20:16:21.353675 936 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.399245 30601 catalog_manager.cc:5495] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } }
W20260806 20:16:21.450160 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 63289.7us
W20260806 20:16:21.467664 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 108064us
I20260806 20:16:21.366086 1019 consensus_queue.cc:1035] T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20260806 20:16:21.355263 1280 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.490562 30601 catalog_manager.cc:5495] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 4 to 5, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 5 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
W20260806 20:16:21.392201 1298 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s user 0.025s sys 0.013s
W20260806 20:16:21.492594 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 156623us
I20260806 20:16:21.496536 30601 catalog_manager.cc:5495] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 reported cstate change: term changed from 4 to 8, leader changed from 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196) to d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194). New cstate: current_term: 8 leader_uuid: "d4181d4d907f477f9feddc44ba3b9207" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 20:16:21.531258 30676 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:21.532524 30676 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:21.533667 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:21.460835 1320 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.466518 1209 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:21.466866 1189 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.451303 1241 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:21.540241 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:21.387914 1322 raft_consensus.cc:2726] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:21.399005 1076 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:21.462659 1124 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.441334 1278 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:21.461256 1275 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:21.491159 1327 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.467376 1051 leader_election.cc:288] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:21.549270 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:21.562312 31421 raft_consensus.cc:1226] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:21.663734 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 171243us
I20260806 20:16:21.437726 1041 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.543928 969 raft_consensus.cc:3032] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:21.466895 1292 consensus_queue.cc:1035] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.193s
I20260806 20:16:21.460255 1310 consensus_queue.cc:1046] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.195s
I20260806 20:16:21.547704 1298 raft_consensus.cc:2726] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 20:16:21.566354 1189 raft_consensus.cc:2781] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:21.720240 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:21.688879 30713 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 20:16:21.692281 31420 raft_consensus.cc:2379] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:21.692509 30757 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:21.568485 1282 consensus_meta.cc:287] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.556s user 0.014s sys 0.000s
I20260806 20:16:21.707434 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 397) took 1220 ms. Trace:
I20260806 20:16:21.723877 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:21.709945 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 357843us
I20260806 20:16:21.563889 1293 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.563539 987 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.730252 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:21.681154 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 148701us
I20260806 20:16:21.731846 30870 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260806 20:16:21.732621 30676 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:21.723240 30710 rpcz_store.cc:276] 0806 20:16:20.487078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:21.080697 (+593619us) service_pool.cc:227] Handling call
0806 20:16:21.080956 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:21.081152 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:16:21.081585 (+ 433us) log.cc:842] Serialized 14 byte log entry
0806 20:16:21.706898 (+625313us) log.cc:842] Serialized 14 byte log entry
0806 20:16:21.707060 (+ 162us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:21.707106 (+ 46us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:21.707157 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:21.707238 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:21.707369 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32970,"prepare.queue_time_us":4895,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"spinlock_wait_cycles":187392,"thread_start_us":590794,"threads_started":2,"wal-append.queue_time_us":629827}
I20260806 20:16:21.739183 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:21.740417 30710 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:21.709537 1076 raft_consensus.cc:3032] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 20:16:21.743619 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:21.682948 1322 raft_consensus.cc:482] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:21.747773 30865 raft_consensus.cc:2454] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:21.706341 969 raft_consensus.cc:729] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:21.708997 1051 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:21.717011 942 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.565893 1241 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.731676 1189 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:21.567276 1307 raft_consensus.cc:504] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.729971 1040 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:21.750053 30835 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.194:42029 timed out after 3.238s (SENT) [suppressed 5 similar messages]
I20260806 20:16:21.691085 1278 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.747426 1076 raft_consensus.cc:729] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:21.546907 1320 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.744979 1019 raft_consensus.cc:482] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:21.787721 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:21.788368 30787 raft_consensus.cc:1226] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260806 20:16:21.789181 1241 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.789522 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:21.790823 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:21.791141 1195 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.763531 1051 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.792639 1321 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.793174 1321 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.793838 1051 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:21.794296 1345 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:21.794754 1345 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:21.795087 1345 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:21.795468 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:21.771687 1189 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:21.549134 1303 consensus_queue.cc:237] T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.699615 1327 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.787859 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:16:21.794025 1317 log.cc:924] Time spent T 47b5901783b8423c9e494c131b0a64e9 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.423s user 0.000s sys 0.001s
I20260806 20:16:21.735450 1282 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.735801 1293 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.804406 30787 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:21.796088 30792 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
W20260806 20:16:21.783075 30783 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1967) took 3382 ms (client timeout 3238 ms). Trace:
W20260806 20:16:21.816485 30783 rpcz_store.cc:269] 0806 20:16:18.400367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.933590 (+533223us) service_pool.cc:227] Handling call
0806 20:16:21.782874 (+2849284us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b540026f054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:21.782994 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2054145280}
I20260806 20:16:21.739014 987 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.805536 30756 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:21.784989 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:21.795869 936 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:21.820662 30789 raft_consensus.cc:2379] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:21.784366 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:16:21.809969 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 60004.1us
I20260806 20:16:21.824036 30869 raft_consensus.cc:3037] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:21.776655 942 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.831562 942 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.834044 942 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.776078 1339 raft_consensus.cc:2781] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 20:16:21.835666 1279 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:21.903302 1295 consensus_meta.cc:287] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 0.574s user 0.009s sys 0.000s
I20260806 20:16:21.903810 1295 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:21.905912 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 82517.8us
I20260806 20:16:21.774840 1258 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.790753 1030 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.821611 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:21.814868 1292 raft_consensus.cc:482] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:21.796051 1353 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:21.825565 1343 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 20:16:21.826500 1340 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
I20260806 20:16:21.956254 30869 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 5.
W20260806 20:16:21.826860 1335 log.cc:924] Time spent T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.385s user 0.000s sys 0.001s
I20260806 20:16:21.827134 1337 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:21.827688 1323 raft_consensus.cc:482] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:21.830711 1321 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:21.805094 1255 leader_election.cc:288] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:21.797060 1310 raft_consensus.cc:3032] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:16:21.974385 1310 raft_consensus.cc:729] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:21.975574 1310 consensus_queue.cc:260] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.977531 1310 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:21.776219 1298 consensus_queue.cc:1046] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260806 20:16:21.978500 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:21.978866 1298 raft_consensus.cc:3032] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:21.979323 1298 raft_consensus.cc:729] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:21.980036 31420 raft_consensus.cc:2379] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 4: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:21.980365 1298 consensus_queue.cc:260] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.909319 30711 raft_consensus.cc:1226] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 9. Current term is 10. Ops: []
I20260806 20:16:21.909677 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:21.982681 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:21.911646 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:21.914520 30865 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 20:16:21.982933 1195 consensus_queue.cc:1159] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "42cabca48db7403b8523a3382a046043" responder_term: 6 status { last_received { term: 6 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 6 index: 7 } }
W20260806 20:16:21.911931 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 219504us
I20260806 20:16:21.979149 1344 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:21.987686 1344 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.837133 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:21.919518 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:21.920228 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:21.762112 1322 raft_consensus.cc:504] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.822834 1241 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:21.963542 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1575) took 1674 ms (client timeout 2069 ms). Trace:
W20260806 20:16:22.006963 30869 rpcz_store.cc:269] 0806 20:16:20.282580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.282843 (+ 263us) service_pool.cc:227] Handling call
0806 20:16:21.784215 (+1501372us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:16:21.956871 (+172656us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28092288}
I20260806 20:16:21.820152 1350 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:21.912478 942 raft_consensus.cc:2781] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election won for term 8
W20260806 20:16:21.968670 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 236128us
I20260806 20:16:21.965837 1041 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:21.968693 1337 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.812747 1019 raft_consensus.cc:504] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:21.977998 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:21.991882 31421 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 10.
I20260806 20:16:22.088424 30676 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:21.987705 1262 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:21.992235 30789 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:21.992487 30792 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:22.096691 1156 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:22.097318 1156 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:22.097796 1343 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:22.099483 1156 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:22.099977 1346 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:22.100517 1346 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:22.006891 878 raft_consensus.cc:3032] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:22.101816 31418 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:21.909785 1076 consensus_queue.cc:260] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.991655 30787 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:22.106020 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1599) took 2679 ms. Trace:
I20260806 20:16:22.004952 30710 raft_consensus.cc:2454] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9.
I20260806 20:16:22.106465 30706 rpcz_store.cc:276] 0806 20:16:19.426398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:19.553992 (+127594us) service_pool.cc:227] Handling call
0806 20:16:19.639140 (+ 85148us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:20.117772 (+478632us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:22.105738 (+1987966us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b5400270e54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:22.105932 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1073059584}
I20260806 20:16:22.107364 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:22.107569 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:22.108006 30710 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:22.108850 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:21.913897 1280 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:22.109216 1348 raft_consensus.cc:2781] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 20:16:22.109757 1348 raft_consensus.cc:482] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
W20260806 20:16:22.110731 986 consensus_meta.cc:287] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.840s user 0.010s sys 0.000s
I20260806 20:16:22.110860 31415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48918 (request call id 443) took 3078 ms. Trace:
I20260806 20:16:22.111423 878 raft_consensus.cc:729] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
W20260806 20:16:22.112329 30790 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1513) took 1677 ms (client timeout 2069 ms). Trace:
W20260806 20:16:22.112751 30790 rpcz_store.cc:269] 0806 20:16:20.434876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.435011 (+ 135us) service_pool.cc:227] Handling call
0806 20:16:22.112125 (+1677114us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b5400002b54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:22.112240 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1384256384}
I20260806 20:16:22.003129 1323 raft_consensus.cc:504] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:22.114061 30790 raft_consensus.cc:1226] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:22.114269 1204 raft_consensus.cc:482] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:22.114662 1204 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:22.115144 1323 leader_election.cc:288] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:22.116346 1327 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:22.097011 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:22.164016 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:22.083220 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:22.092792 30832 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:21.809140 969 consensus_queue.cc:260] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:22.094398 30786 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:22.169059 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 76370.5us
I20260806 20:16:22.101591 1356 consensus_queue.cc:1046] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.131s
I20260806 20:16:21.963025 1353 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:21.965890 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:22.013370 1298 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:22.110129 1348 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:22.111249 31415 rpcz_store.cc:276] 0806 20:16:19.031997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:19.041679 (+ 9682us) service_pool.cc:227] Handling call
0806 20:16:20.053251 (+1011572us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:20.098525 (+ 45274us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:22.110002 (+2011477us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b54005d39d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:22.110665 (+ 663us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b54005d39d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:22.110803 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2287137792}
I20260806 20:16:22.115381 30790 raft_consensus.cc:1226] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:22.014897 1295 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:21.959499 1339 raft_consensus.cc:686] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
W20260806 20:16:21.992996 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 248969us
I20260806 20:16:22.111040 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:22.202328 30834 leader_election.cc:302] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:22.065268 1226 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:22.176703 30756 leader_election.cc:302] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:22.177282 30783 raft_consensus.cc:2379] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 5: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:22.208145 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1983) took 2471 ms. Trace:
I20260806 20:16:22.113111 1322 leader_election.cc:288] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:22.208531 30791 rpcz_store.cc:276] 0806 20:16:19.737146 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:19.737329 (+ 183us) service_pool.cc:227] Handling call
0806 20:16:19.811537 (+ 74208us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:19.865768 (+ 54231us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:22.201899 (+2336131us) spinlock_profiling.cc:243] Waited 53.3 ms on lock 0x7b5400260554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:22.208062 (+ 6163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122529024}
I20260806 20:16:22.177605 30789 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:22.108210 30706 raft_consensus.cc:2454] T 1ee7927d05584c999f6e0a2b753336f1 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
W20260806 20:16:22.179728 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 91398.8us
I20260806 20:16:22.175769 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 399) took 1066 ms. Trace:
I20260806 20:16:22.082299 1120 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:22.110069 1366 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:22.214442 30676 leader_election.cc:302] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:22.216017 1346 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:22.214373 30705 rpcz_store.cc:276] 0806 20:16:21.109145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:21.289926 (+180781us) service_pool.cc:227] Handling call
0806 20:16:21.342242 (+ 52316us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:21.493739 (+151497us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:21.493769 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:21.532673 (+ 38904us) log.cc:842] Serialized 26 byte log entry
0806 20:16:21.772487 (+239814us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:21.772526 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:21.772591 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:22.175550 (+402959us) raft_consensus.cc:1671] finished
0806 20:16:22.175582 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:22.175714 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":830,"raft.queue_time_us":347518,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":33094,"thread_start_us":240284,"threads_started":2,"wal-append.queue_time_us":381302}
I20260806 20:16:22.290707 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:22.278587 30655 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 58d59beb597f4ed99d84d102ddb41ff5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } }
I20260806 20:16:22.303534 30870 raft_consensus.cc:2396] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260806 20:16:22.216323 1019 leader_election.cc:288] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:22.217109 1320 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:22.217494 1342 log.cc:924] Time spent T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
I20260806 20:16:22.280218 30756 leader_election.cc:302] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:22.281025 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:22.352492 30790 raft_consensus.cc:2379] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 8: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:22.353829 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:22.354529 30870 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:21.960300 1292 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:22.284643 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 71510.1us
I20260806 20:16:22.440850 31389 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:22.198374 1280 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:22.340358 30791 raft_consensus.cc:1261] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 20:16:22.341037 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:22.218571 1156 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:22.085477 1350 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:22.220757 1226 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:22.112347 878 consensus_queue.cc:260] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:22.427251 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 209114us
I20260806 20:16:22.427853 1346 raft_consensus.cc:2781] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 20:16:22.207391 1209 consensus_queue.cc:1159] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "58d59beb597f4ed99d84d102ddb41ff5" responder_term: 6 status { last_received { term: 6 index: 5 } last_committed_idx: 3 last_received_current_leader { term: 6 index: 5 } }
W20260806 20:16:22.286597 1012 consensus_meta.cc:287] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.932s user 0.012s sys 0.005s
I20260806 20:16:22.446607 30706 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 8: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:22.287475 1327 raft_consensus.cc:2781] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:16:22.176740 986 raft_consensus.cc:504] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:22.289391 1295 raft_consensus.cc:2726] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:22.429471 1309 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260806 20:16:22.452914 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 382907us
I20260806 20:16:22.294637 1307 leader_election.cc:288] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:22.352067 1343 raft_consensus.cc:2781] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:22.455894 1044 raft_consensus.cc:2781] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:22.515133 1044 raft_consensus.cc:482] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42cabca48db7403b8523a3382a046043)
I20260806 20:16:22.515588 1044 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:22.496289 1012 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:22.463529 1346 raft_consensus.cc:482] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:22.513691 986 leader_election.cc:288] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:22.518136 1346 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:22.514343 30710 raft_consensus.cc:2379] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 8: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:22.497300 1343 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:22.519345 1343 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:22.518343 1365 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.300s user 0.000s sys 0.000s
I20260806 20:16:22.288338 1356 consensus_queue.cc:1046] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.177s
I20260806 20:16:22.311261 1120 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:22.465322 1293 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:22.496654 1320 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:22.496904 1309 raft_consensus.cc:482] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:22.452589 1280 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:22.456667 1156 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:22.784750 29419 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 20:16:22.514597 31418 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:23.098476 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:22.816635 30870 raft_consensus.cc:2454] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
W20260806 20:16:23.096896 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 882493us
W20260806 20:16:22.784468 30757 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.096s sys 0.204s
W20260806 20:16:23.128230 30757 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.120s user 0.099s sys 0.214s
W20260806 20:16:23.128533 30757 connection.cc:232] RPC call timeout handler was delayed by -2.0630884149868507s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260806 20:16:23.097689 30783 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.006s sys 0.018s
W20260806 20:16:23.130539 30783 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.811s user 0.009s sys 0.021s
I20260806 20:16:23.130401 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:23.053088 30786 consensus_meta.cc:287] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.875s user 0.007s sys 0.000s
W20260806 20:16:23.013921 942 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.838s user 0.003s sys 0.014s
I20260806 20:16:22.584017 1327 raft_consensus.cc:686] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 9 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:23.132334 30786 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
W20260806 20:16:23.095719 1343 consensus_meta.cc:287] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 0.576s user 0.008s sys 0.000s
I20260806 20:16:23.109462 30787 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 2000) took 1307 ms. Trace:
I20260806 20:16:23.171593 30787 rpcz_store.cc:276] 0806 20:16:21.801591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:22.211954 (+410363us) service_pool.cc:227] Handling call
0806 20:16:23.108901 (+896947us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:23.109108 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:23.109134 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:23.109174 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:23.109207 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:23.109262 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:23.109375 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:23.132730 30709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41700 (request call id 383) took 4151 ms (client timeout 4412 ms). Trace:
I20260806 20:16:23.172183 30792 raft_consensus.cc:1261] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260806 20:16:22.886870 1367 log.cc:924] Time spent T 672b273ef36e4c6eb6873cc61471cb53 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.442s user 0.000s sys 0.000s
W20260806 20:16:23.172631 30709 rpcz_store.cc:269] 0806 20:16:18.974647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:19.061755 (+ 87108us) service_pool.cc:227] Handling call
0806 20:16:23.126186 (+4064431us) spinlock_profiling.cc:243] Waited 41 ms on lock 0x7b5400175054. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:23.126323 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94255360}
I20260806 20:16:23.128887 30676 leader_election.cc:302] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:23.131639 30870 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:22.605556 1295 raft_consensus.cc:482] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:22.607605 1282 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 10 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:23.174502 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:23.092525 1292 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:23.175719 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 47035.8us
I20260806 20:16:23.096581 1120 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:23.102860 1019 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:22.886196 1368 log.cc:924] Time spent T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.545s user 0.001s sys 0.000s
I20260806 20:16:23.191923 30676 leader_election.cc:302] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:23.177349 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:23.167601 942 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.000s user 0.009s sys 0.017s
I20260806 20:16:23.102967 1309 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Advancing to term 11
I20260806 20:16:23.102687 1278 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:23.127286 987 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:23.102587 969 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:23.169054 1310 consensus_meta.cc:287] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.191s user 0.008s sys 0.004s
I20260806 20:16:23.196122 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:23.210649 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:23.060906 1297 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.848s
I20260806 20:16:23.205809 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:23.206481 30757 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:23.130839 1353 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:23.214522 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:23.182570 986 raft_consensus.cc:2781] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:23.219901 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:23.164134 31418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 554) took 1173 ms. Trace:
I20260806 20:16:23.221992 30757 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
I20260806 20:16:23.222074 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:22.885016 1363 consensus_queue.cc:1046] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.755s
I20260806 20:16:23.250159 30870 raft_consensus.cc:3037] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:23.223614 30676 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:23.192320 1356 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.192754 1372 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.257670 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:23.258780 30792 raft_consensus.cc:2454] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 6.
I20260806 20:16:23.260020 30757 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043
W20260806 20:16:23.268334 30676 connection.cc:232] RPC call timeout handler was delayed by -2.9856763900024816s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260806 20:16:23.266834 30870 raft_consensus.cc:2454] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:23.201918 942 raft_consensus.cc:686] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:22.342232 1366 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:23.214407 1120 raft_consensus.cc:2781] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:23.220675 30709 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
W20260806 20:16:23.270710 31389 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.824s user 0.009s sys 0.023s
I20260806 20:16:23.271483 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf7304ca0290443b884297d0e8744607" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:23.271088 30756 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.000s sys 0.020s
I20260806 20:16:23.180603 1343 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.128988 1320 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:22.606407 1339 consensus_queue.cc:237] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:23.221387 1226 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:23.223300 986 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:23.199580 1295 raft_consensus.cc:504] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.203030 1019 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:23.313366 31387 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.230s user 0.062s sys 0.123s
I20260806 20:16:23.246558 1310 raft_consensus.cc:2726] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 20:16:23.228677 1350 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:23.668712 31389 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.222s user 0.016s sys 0.054s
I20260806 20:16:23.669852 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1789) took 1458 ms. Trace:
I20260806 20:16:23.363161 30792 raft_consensus.cc:1261] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260806 20:16:23.443988 30783 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 2008) took 1339 ms. Trace:
W20260806 20:16:23.572868 31388 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.091s user 0.024s sys 0.018s
W20260806 20:16:23.671730 31388 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.190s user 0.024s sys 0.027s
I20260806 20:16:23.672636 31388 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
W20260806 20:16:23.669337 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 1.22275e+06us
I20260806 20:16:23.428323 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:23.320314 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
W20260806 20:16:23.674904 31389 connection.cc:232] RPC call timeout handler was delayed by -1.4587806880008429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23250000000000004s for a response
I20260806 20:16:23.426488 31417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49046 (request call id 644) took 2947 ms. Trace:
W20260806 20:16:23.673588 31387 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.591s user 0.066s sys 0.166s
I20260806 20:16:23.673892 30709 rpcz_store.cc:276] 0806 20:16:22.211294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.202605 (+991311us) service_pool.cc:227] Handling call
0806 20:16:23.669788 (+467183us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:23.319789 30756 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.559s user 0.000s sys 0.024s
W20260806 20:16:23.676923 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 916679us
I20260806 20:16:23.675693 31417 rpcz_store.cc:276] 0806 20:16:20.479471 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.479654 (+ 183us) service_pool.cc:227] Handling call
0806 20:16:20.559059 (+ 79405us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:21.096341 (+537282us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:21.096374 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:21.096417 (+ 43us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:21.096447 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:23.426217 (+2329770us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54005ed8d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:23.426266 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:23.426421 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1235132800}
I20260806 20:16:23.676437 30790 raft_consensus.cc:2362] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7. Re-sending same reply.
I20260806 20:16:23.418269 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 258) took 1674 ms. Trace:
I20260806 20:16:23.670007 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1487) took 1198 ms. Trace:
I20260806 20:16:23.681676 30870 rpcz_store.cc:276] 0806 20:16:22.471551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.271068 (+799517us) service_pool.cc:227] Handling call
0806 20:16:23.669948 (+398880us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:23.681440 30865 rpcz_store.cc:276] 0806 20:16:21.743506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:21.768360 (+ 24854us) service_pool.cc:227] Handling call
0806 20:16:21.768693 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:21.983310 (+214617us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:23.417957 (+1434647us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b5400149e54. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:23.418193 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":331400064}
I20260806 20:16:23.683337 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:23.683655 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:23.256896 1296 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.013s sys 0.007s
I20260806 20:16:23.670853 30783 rpcz_store.cc:276] 0806 20:16:22.104200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:22.206660 (+102460us) service_pool.cc:227] Handling call
0806 20:16:22.206960 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:22.210022 (+ 3062us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:22.210053 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:22.221615 (+ 11562us) log.cc:842] Serialized 26 byte log entry
0806 20:16:23.131069 (+909454us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:23.131118 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:23.131183 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:23.443693 (+312510us) raft_consensus.cc:1671] finished
0806 20:16:23.443715 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:23.443906 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2643,"raft.queue_time_us":7929,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":811930,"threads_started":1,"wal-append.queue_time_us":1123026}
W20260806 20:16:23.688704 30832 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.168s user 0.004s sys 0.060s
W20260806 20:16:23.690625 30835 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.003s sys 0.007s
I20260806 20:16:23.691429 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:23.691181 30832 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.170s user 0.005s sys 0.060s
W20260806 20:16:23.692160 30835 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.003s sys 0.007s
I20260806 20:16:23.258817 1282 raft_consensus.cc:2781] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 20:16:23.645282 1204 consensus_meta.cc:287] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.530s user 0.007s sys 0.004s
W20260806 20:16:23.559749 1298 consensus_meta.cc:287] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.346s user 0.005s sys 0.001s
W20260806 20:16:23.460675 1189 consensus_meta.cc:287] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.640s user 0.012s sys 0.000s
W20260806 20:16:23.449072 1348 consensus_meta.cc:287] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.241s user 0.004s sys 0.004s
W20260806 20:16:23.456462 1262 consensus_meta.cc:287] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.294s user 0.007s sys 0.003s
W20260806 20:16:23.459244 1044 consensus_meta.cc:287] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.943s user 0.010s sys 0.000s
I20260806 20:16:23.696415 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:23.662308 1241 consensus_meta.cc:287] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.652s user 0.006s sys 0.006s
W20260806 20:16:23.277374 1345 consensus_meta.cc:287] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.482s user 0.012s sys 0.000s
I20260806 20:16:23.313535 1353 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:23.274577 1278 raft_consensus.cc:2781] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election won for term 8
W20260806 20:16:23.688503 1369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.024s sys 0.045s
I20260806 20:16:23.703127 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:23.431283 1377 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:23.425925 1328 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.668881 986 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:23.732987 1377 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.735167 1377 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:23.735864 30870 raft_consensus.cc:3037] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:23.707314 1282 raft_consensus.cc:686] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [term 5 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:16:23.690409 1296 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.261s user 0.025s sys 0.016s
I20260806 20:16:23.706523 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:23.669629 1120 raft_consensus.cc:482] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:23.423019 1277 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:23.680646 1294 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.451045 1280 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:23.708070 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:23.718601 1353 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:23.709137 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:23.816370 30792 raft_consensus.cc:2379] T 3070a7930e6d47cd8776055aab6539a3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:23.711452 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:23.695034 1226 raft_consensus.cc:2741] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 20:16:23.685428 1350 raft_consensus.cc:482] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:23.690620 1310 raft_consensus.cc:482] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.712482 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f32336173df4611829be37d396e86ba" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:23.708755 1189 raft_consensus.cc:504] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.713709 30709 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:23.837674 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1629) took 1379 ms. Trace:
I20260806 20:16:23.249507 31418 rpcz_store.cc:276] 0806 20:16:21.960994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:21.963789 (+ 2795us) service_pool.cc:227] Handling call
0806 20:16:23.134537 (+1170748us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:23.279270 1356 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.761026 30869 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9.
I20260806 20:16:23.771677 30870 raft_consensus.cc:2454] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9.
I20260806 20:16:23.845202 30870 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1488) took 1336 ms. Trace:
I20260806 20:16:23.845580 30870 rpcz_store.cc:276] 0806 20:16:22.509144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.702834 (+1193690us) service_pool.cc:227] Handling call
0806 20:16:23.845127 (+142293us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:23.732206 1369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s user 0.024s sys 0.046s
I20260806 20:16:23.450235 1374 raft_consensus.cc:482] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:16:23.793561 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 119316us
W20260806 20:16:23.812379 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 110865us
W20260806 20:16:23.812947 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 120956us
I20260806 20:16:23.710992 1348 raft_consensus.cc:504] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:23.274619 1372 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.719485 1298 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:23.552047 942 consensus_queue.cc:237] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.364473 1327 consensus_queue.cc:237] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.818606 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:23.762717 1328 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:23.820119 30833 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.092s user 0.000s sys 0.046s
I20260806 20:16:23.932802 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:23.933247 30833 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.343s user 0.000s sys 0.046s
I20260806 20:16:23.721851 1270 raft_consensus.cc:482] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:23.964983 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:23.823760 1282 consensus_queue.cc:237] T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.824215 1019 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:23.829576 1294 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.843709 1350 raft_consensus.cc:504] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.832643 1395 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.976198 1398 raft_consensus.cc:482] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.837991 30709 rpcz_store.cc:276] 0806 20:16:22.457992 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.676863 (+1218871us) service_pool.cc:227] Handling call
0806 20:16:23.837615 (+160752us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:23.730520 1021 raft_consensus.cc:3032] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Stepping down as leader of term 7
W20260806 20:16:23.838658 30707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1611) took 3367 ms (client timeout 2341 ms). Trace:
W20260806 20:16:24.148440 30707 rpcz_store.cc:269] 0806 20:16:20.471636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.483877 (+ 12241us) service_pool.cc:227] Handling call
0806 20:16:23.774741 (+3290864us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54001b5654. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:23.838515 (+ 63774us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1233268992}
I20260806 20:16:23.756908 1377 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.721812 1241 raft_consensus.cc:2726] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:16:23.833346 969 consensus_meta.cc:287] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.620s user 0.005s sys 0.001s
I20260806 20:16:23.874888 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:23.836905 1353 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:23.722079 1345 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:23.718569 1044 raft_consensus.cc:504] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:23.928558 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 50403.6us
I20260806 20:16:23.966176 31415 raft_consensus.cc:1226] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 20:16:23.966066 31389 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:23.964918 30713 raft_consensus.cc:2379] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 10: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:23.973721 1270 raft_consensus.cc:504] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:23.974787 1393 raft_consensus.cc:482] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:23.703328 1204 raft_consensus.cc:504] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:23.976159 30819 heartbeater.cc:644] Failed to heartbeat to 127.28.186.254:40443 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.28.186.254:40443 timed out after 15.000s (SENT)
I20260806 20:16:23.977392 1189 leader_election.cc:288] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:23.982015 1382 raft_consensus.cc:482] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.834738 1387 consensus_queue.cc:1046] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.192s
I20260806 20:16:23.983443 30834 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:23.966887 1391 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:23.965559 1389 raft_consensus.cc:482] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:23.932328 1394 consensus_queue.cc:1035] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
W20260806 20:16:23.969110 1346 consensus_meta.cc:287] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.872s user 0.005s sys 0.004s
I20260806 20:16:23.787606 1120 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:23.846613 1293 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:23.945417 30708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41700 (request call id 380) took 5111 ms (client timeout 3248 ms). Trace:
I20260806 20:16:24.059296 1401 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:24.077298 30757 leader_election.cc:302] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:23.928918 1255 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:20.062439 30784 rpcz_store.cc:276] 0806 20:16:18.747202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:19.302254 (+555052us) service_pool.cc:227] Handling call
0806 20:16:19.302528 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:19.342202 (+ 39674us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 20:16:19.804048 (+461846us) log.cc:842] Serialized 14 byte log entry
0806 20:16:20.055619 (+251571us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:20.055672 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:20.055722 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:20.055795 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:20.055902 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":932,"prepare.queue_time_us":699354,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":914,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":3766,"thread_start_us":621827,"threads_started":2}
W20260806 20:16:24.079946 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 377512us
I20260806 20:16:23.930900 1343 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:23.816522 1366 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:23.843797 1310 raft_consensus.cc:504] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:23.833973 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1571) took 3681 ms. Trace:
I20260806 20:16:23.875489 1277 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.465441 30835 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:24.023211 1298 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.264703 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:24.466492 30709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1795) took 1294 ms (client timeout 1525 ms). Trace:
I20260806 20:16:23.893873 1374 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:24.434906 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 451522us
I20260806 20:16:24.279989 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:24.438513 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 361302us
W20260806 20:16:24.439464 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 473474us
I20260806 20:16:24.440793 30787 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:24.434506 30789 raft_consensus.cc:2379] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:16:24.434240 30708 rpcz_store.cc:269] 0806 20:16:18.817573 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:18.853528 (+ 35955us) service_pool.cc:227] Handling call
0806 20:16:23.902217 (+5048689us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003bb0d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:23.928799 (+ 26582us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2630571776}
I20260806 20:16:24.083595 1254 raft_consensus.cc:3032] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:24.086431 1278 raft_consensus.cc:686] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:24.460086 1343 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:24.461432 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:24.464607 30868 rpcz_store.cc:276] 0806 20:16:20.152876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.153005 (+ 129us) service_pool.cc:227] Handling call
0806 20:16:20.153309 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:20.183102 (+ 29793us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:23.833708 (+3650606us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b540027fbd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:23.833888 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41560448}
I20260806 20:16:24.081072 1226 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:24.253137 1021 raft_consensus.cc:729] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:24.080335 1206 consensus_queue.cc:260] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:24.518489 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:24.519918 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:24.520678 30757 leader_election.cc:302] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:24.445258 1390 raft_consensus.cc:2726] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:24.522446 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:24.513747 30707 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
W20260806 20:16:24.539618 30707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1619) took 2814 ms (client timeout 2992 ms). Trace:
W20260806 20:16:24.540048 30707 rpcz_store.cc:269] 0806 20:16:21.725440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.279623 (+2554183us) service_pool.cc:227] Handling call
0806 20:16:24.539559 (+259936us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.541425 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:24.535023 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:24.542243 30784 raft_consensus.cc:2454] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:24.535784 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1508) took 1210 ms. Trace:
I20260806 20:16:24.542814 30868 rpcz_store.cc:276] 0806 20:16:23.325182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.519629 (+1194447us) service_pool.cc:227] Handling call
0806 20:16:24.535723 (+ 16094us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.543643 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:24.466904 30709 rpcz_store.cc:269] 0806 20:16:23.172059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.148298 (+976239us) service_pool.cc:227] Handling call
0806 20:16:24.466429 (+318131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.536566 30757 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:24.545413 30708 raft_consensus.cc:2379] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
W20260806 20:16:24.545881 30708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1626) took 2460 ms (client timeout 2906 ms). Trace:
W20260806 20:16:24.546314 30708 rpcz_store.cc:269] 0806 20:16:22.085147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.522157 (+2437010us) service_pool.cc:227] Handling call
0806 20:16:24.545824 (+ 23667us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.536936 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:24.537649 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1573) took 4343 ms. Trace:
I20260806 20:16:24.448120 1262 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 20:16:24.527029 30703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1767) took 4417 ms (client timeout 3046 ms). Trace:
I20260806 20:16:24.547173 30861 rpcz_store.cc:276] 0806 20:16:20.183615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.183760 (+ 145us) service_pool.cc:227] Handling call
0806 20:16:20.398795 (+215035us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:20.405370 (+ 6575us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:24.526979 (+4121609us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002fadd0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:24.527105 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":405876608}
I20260806 20:16:24.547389 30790 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
W20260806 20:16:24.550712 30703 rpcz_store.cc:269] 0806 20:16:20.109395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.109521 (+ 126us) service_pool.cc:227] Handling call
0806 20:16:24.526694 (+4417173us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54000a87d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:24.526944 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":691552128}
I20260806 20:16:24.552105 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:24.548787 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1507) took 1272 ms. Trace:
I20260806 20:16:24.539436 30792 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:24.547878 30789 raft_consensus.cc:2379] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 3: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:24.555162 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:24.555250 30868 rpcz_store.cc:276] 0806 20:16:23.276936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.543416 (+1266480us) service_pool.cc:227] Handling call
0806 20:16:24.548731 (+ 5315us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.556088 30784 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 9: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:24.513404 1254 raft_consensus.cc:729] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:24.558786 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:24.438303 1346 raft_consensus.cc:504] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.558889 30703 raft_consensus.cc:2396] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260806 20:16:24.560390 30757 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:24.563338 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 96358.9us
I20260806 20:16:24.253849 1395 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.448396 1401 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.577891 1395 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.559532 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:24.284409 1377 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:24.579206 1401 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.580196 1377 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:24.581077 30789 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:24.581535 1294 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:24.582119 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:24.569481 1021 consensus_queue.cc:260] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.583890 30789 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:24.584156 1021 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:24.584297 1044 leader_election.cc:288] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:24.585677 1388 log.cc:924] Time spent T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.181s user 0.001s sys 0.000s
I20260806 20:16:24.586966 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1606) took 4167 ms. Trace:
W20260806 20:16:24.587590 1383 log.cc:924] Time spent T 00000000000000000000000000000000 P 25d2453323cc4a96886081ec67cef6f6: Append to log took a long time: real 0.657s user 0.001s sys 0.000s
W20260806 20:16:24.588490 1025 consensus_queue.cc:1159] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d4181d4d907f477f9feddc44ba3b9207" responder_term: 8 status { last_received { term: 7 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 20:16:24.569950 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:24.533358 1226 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:24.590241 1310 leader_election.cc:288] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:24.540544 1366 raft_consensus.cc:2726] T bf7304ca0290443b884297d0e8744607 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:24.591599 1226 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.592772 1226 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 20:16:24.570188 1254 consensus_queue.cc:260] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:24.593490 30789 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:24.593696 1372 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.594035 1254 raft_consensus.cc:3037] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:24.594249 1400 raft_consensus.cc:2781] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:16:24.571242 30868 raft_consensus.cc:2379] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:24.594456 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:24.594863 1400 raft_consensus.cc:686] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [term 9 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:24.595039 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1503) took 1376 ms. Trace:
I20260806 20:16:24.595227 30789 raft_consensus.cc:2379] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:24.595460 30868 rpcz_store.cc:276] 0806 20:16:23.219107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.558525 (+1339418us) service_pool.cc:227] Handling call
0806 20:16:24.594974 (+ 36449us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.596439 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:24.596946 30868 raft_consensus.cc:3037] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:24.553135 1320 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:24.601150 1277 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.567014 1403 raft_consensus.cc:482] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:24.601919 1403 raft_consensus.cc:504] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.545670 1295 leader_election.cc:288] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:24.604051 1403 leader_election.cc:288] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.549151 1262 raft_consensus.cc:482] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:24.604579 1262 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:24.552727 1206 raft_consensus.cc:3037] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:24.436830 1391 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.566716 1390 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:24.583326 30835 leader_election.cc:302] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:24.612212 30868 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:24.612880 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:24.613276 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:24.613812 30703 raft_consensus.cc:2454] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:24.614331 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1794) took 1442 ms. Trace:
I20260806 20:16:24.614638 30703 rpcz_store.cc:276] 0806 20:16:23.171765 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.612637 (+1440872us) service_pool.cc:227] Handling call
0806 20:16:24.614277 (+ 1640us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.614086 30868 raft_consensus.cc:2379] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:24.446969 1293 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.587299 30712 rpcz_store.cc:276] 0806 20:16:20.419725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.419846 (+ 121us) service_pool.cc:227] Handling call
0806 20:16:20.474017 (+ 54171us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:20.474407 (+ 390us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:20.474435 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:23.428584 (+2954149us) spinlock_profiling.cc:243] Waited 8.78 ms on lock 0x7b2c00055798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f715005ee11 00007f715006e442 00007f71516aac11 00007f715166f58a 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772
0806 20:16:23.429021 (+ 437us) log.cc:842] Serialized 26 byte log entry
0806 20:16:23.722034 (+293013us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:23.722095 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:23.722188 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:24.586664 (+864476us) raft_consensus.cc:1671] finished
0806 20:16:24.586688 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:24.586905 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":109,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":24810,"spinlock_wait_cycles":20198784,"thread_start_us":292358,"threads_started":1,"wal-append.queue_time_us":629304}
I20260806 20:16:24.588506 30787 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:24.611020 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:24.574218 1356 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.265998 1398 raft_consensus.cc:504] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.635012 30835 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:24.615512 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:24.553391 1322 raft_consensus.cc:482] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:24.617795 30709 raft_consensus.cc:2396] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 20:16:24.615340 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:24.688982 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1801) took 1070 ms. Trace:
I20260806 20:16:24.689358 30709 rpcz_store.cc:276] 0806 20:16:23.618695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.551736 (+933041us) service_pool.cc:227] Handling call
0806 20:16:24.688907 (+137171us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:24.688764 31387 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.004s sys 0.004s
W20260806 20:16:24.690130 31387 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.760s user 0.004s sys 0.004s
I20260806 20:16:24.690469 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 344) took 1035 ms. Trace:
I20260806 20:16:24.460764 1255 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:24.691277 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:24.691576 1411 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:24.692436 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:24.690467 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:24.693292 30709 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:24.696630 30861 raft_consensus.cc:2454] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 10.
I20260806 20:16:24.697780 30784 rpcz_store.cc:276] 0806 20:16:23.655097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.689576 (+1034479us) service_pool.cc:227] Handling call
0806 20:16:24.689950 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:24.690158 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:24.690191 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:24.690237 (+ 46us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:24.690271 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:24.690317 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:24.690414 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.618510 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:24.619339 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:24.622630 961 raft_consensus.cc:3032] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 20:16:24.707031 30712 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7.
I20260806 20:16:24.626780 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1507) took 4551 ms. Trace:
I20260806 20:16:24.709543 30785 rpcz_store.cc:276] 0806 20:16:20.074967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.075091 (+ 124us) service_pool.cc:227] Handling call
0806 20:16:20.094069 (+ 18978us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:20.177290 (+ 83221us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:24.626517 (+4449227us) spinlock_profiling.cc:243] Waited 76.4 ms on lock 0x7b54003b1fd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:24.626693 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175778176}
I20260806 20:16:24.627820 1278 consensus_queue.cc:237] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.629534 1400 consensus_queue.cc:237] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.630527 1391 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.632323 1239 raft_consensus.cc:3037] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:24.682266 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:24.685897 31388 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.037s sys 0.050s
I20260806 20:16:24.668941 30703 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 10.
I20260806 20:16:24.534220 1343 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:24.727533 31388 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.659s user 0.037s sys 0.050s
I20260806 20:16:24.729233 1343 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.633551 1195 raft_consensus.cc:482] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:24.712635 30708 raft_consensus.cc:2379] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
W20260806 20:16:24.697513 1296 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.001s sys 0.006s
W20260806 20:16:24.719084 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 84157.1us
I20260806 20:16:24.446132 1393 raft_consensus.cc:504] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:24.607410 1397 raft_consensus.cc:2726] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:24.614215 1348 leader_election.cc:288] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 10 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:24.725467 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 210372us
I20260806 20:16:24.725997 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 346) took 1043 ms. Trace:
I20260806 20:16:24.634582 1295 raft_consensus.cc:2726] T 616aae135c70498da00a9e8a1a61b1cd P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:24.445179 1389 raft_consensus.cc:504] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.690666 1322 raft_consensus.cc:504] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.727130 30868 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:24.741636 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:24.692385 1255 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:24.711750 961 raft_consensus.cc:729] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:24.450332 1382 raft_consensus.cc:504] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:24.688383 1402 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:24.692097 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:24.692102 1411 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:24.737939 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1796) took 1544 ms. Trace:
I20260806 20:16:24.718123 1254 consensus_queue.cc:1035] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.308s
I20260806 20:16:24.994619 1298 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.819486 31414 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 LEADER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260806 20:16:24.948665 30713 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1623) took 3222 ms (client timeout 1943 ms). Trace:
W20260806 20:16:25.821772 30713 rpcz_store.cc:269] 0806 20:16:21.726670 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:21.970002 (+243332us) service_pool.cc:227] Handling call
0806 20:16:24.948392 (+2978390us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54001b4254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:24.948564 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4129156992}
W20260806 20:16:24.981804 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 196214us
I20260806 20:16:25.823390 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:25.823483 1393 leader_election.cc:288] T 616aae135c70498da00a9e8a1a61b1cd P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:25.824054 30713 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
W20260806 20:16:24.980960 1296 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.047s user 0.001s sys 0.013s
I20260806 20:16:25.367547 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:24.909999 1309 consensus_meta.cc:287] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Time spent flushing consensus metadata: real 1.697s user 0.011s sys 0.001s
W20260806 20:16:24.981297 30868 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1493) took 2372 ms (client timeout 2992 ms). Trace:
I20260806 20:16:24.942505 1353 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:25.828569 1389 leader_election.cc:288] T cc6e5d61a5694983a73c242e91257d86 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.763001 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:25.829413 31416 raft_consensus.cc:2454] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:25.830125 31416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 560) took 1138 ms. Trace:
I20260806 20:16:25.830509 31416 rpcz_store.cc:276] 0806 20:16:24.691801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.721822 (+ 30021us) service_pool.cc:227] Handling call
0806 20:16:25.830064 (+1108242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.008718 1262 raft_consensus.cc:504] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:24.793272 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:24.884250 1320 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:25.477288 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:25.836454 1120 consensus_meta.cc:287] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.388s user 0.003s sys 0.008s
I20260806 20:16:25.836187 1320 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:24.878331 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:24.942683 30786 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 1989) took 3158 ms (client timeout 1591 ms). Trace:
I20260806 20:16:24.984439 1350 leader_election.cc:288] T 6c8d5f5ccc46476aac7a641abae6bdfa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:25.848704 1417 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:25.232858 1156 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:25.835210 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1802) took 2136 ms. Trace:
I20260806 20:16:25.822393 31414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 562) took 1097 ms. Trace:
W20260806 20:16:25.826591 30868 rpcz_store.cc:269] 0806 20:16:22.608580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.681893 (+2073313us) service_pool.cc:227] Handling call
0806 20:16:24.981229 (+299336us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.849922 30712 rpcz_store.cc:276] 0806 20:16:23.698708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.707827 (+1009119us) service_pool.cc:227] Handling call
0806 20:16:25.835141 (+1127314us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:25.851100 30868 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1586) took 1877 ms (client timeout 1877 ms). Trace:
I20260806 20:16:25.837030 31421 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 564) took 1093 ms. Trace:
W20260806 20:16:25.849076 30786 rpcz_store.cc:269] 0806 20:16:21.784318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:21.787611 (+ 3293us) service_pool.cc:227] Handling call
0806 20:16:24.942456 (+3154845us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54001423d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:24.942601 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":653268352}
W20260806 20:16:25.848390 30676 connection.cc:232] RPC call timeout handler was delayed by -1.8328265529999044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
W20260806 20:16:25.852293 30676 connection.cc:232] RPC call timeout handler was delayed by -1.7410265530052129s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260806 20:16:25.852562 30676 connection.cc:232] RPC call timeout handler was delayed by -1.5889265529986005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1795s for a response
W20260806 20:16:25.852818 30676 connection.cc:232] RPC call timeout handler was delayed by -1.4827265530184377s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
W20260806 20:16:25.853081 30676 connection.cc:232] RPC call timeout handler was delayed by -1.3423265530145727s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260806 20:16:25.848459 30674 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.024s sys 0.036s
W20260806 20:16:25.853755 30674 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.589s user 0.024s sys 0.037s
W20260806 20:16:25.825563 30870 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1502) took 2624 ms (client timeout 1729 ms). Trace:
I20260806 20:16:25.836472 30863 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 8: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:25.838523 961 consensus_queue.cc:260] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:25.839938 30871 raft_consensus.cc:1261] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 20:16:25.849496 1417 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:25.853972 30786 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 2018) took 2156 ms. Trace:
I20260806 20:16:25.840031 1402 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:25.855161 1350 raft_consensus.cc:482] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:25.859536 1350 raft_consensus.cc:504] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:25.860713 1402 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:24.950773 30789 raft_consensus.cc:1226] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:24.936179 1270 leader_election.cc:288] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:25.005281 1344 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:25.856108 30870 rpcz_store.cc:269] 0806 20:16:23.200726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.846263 (+645537us) service_pool.cc:227] Handling call
0806 20:16:25.209279 (+1363016us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:16:25.825483 (+616204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25468160}
I20260806 20:16:25.841899 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41700 (request call id 410) took 1160 ms. Trace:
I20260806 20:16:24.786194 1195 raft_consensus.cc:504] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:25.851758 31421 rpcz_store.cc:276] 0806 20:16:24.743252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.792912 (+ 49660us) service_pool.cc:227] Handling call
0806 20:16:25.836725 (+1043813us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:25.856739 30863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1578) took 1885 ms (client timeout 1772 ms). Trace:
W20260806 20:16:25.863481 30863 rpcz_store.cc:269] 0806 20:16:23.971245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.208889 (+1237644us) service_pool.cc:227] Handling call
0806 20:16:25.856659 (+647770us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:25.863924 1239 consensus_meta.cc:287] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 1.122s user 0.010s sys 0.000s
W20260806 20:16:25.864251 30863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1579) took 1892 ms (client timeout 1884 ms). Trace:
I20260806 20:16:25.864078 1195 leader_election.cc:288] T 7e6251f4f1aa48ca869b81256b9fd89a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:25.864593 30863 rpcz_store.cc:269] 0806 20:16:23.971485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.864074 (+1892589us) service_pool.cc:214] Skipping call since client already timed out
0806 20:16:25.864194 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:16:25.865217 30713 rpcz_store.cc:276] 0806 20:16:24.681513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.822893 (+1141380us) service_pool.cc:227] Handling call
0806 20:16:25.841832 (+ 18939us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.861389 1350 leader_election.cc:288] T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:25.866317 30870 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1588) took 1782 ms (client timeout 1770 ms). Trace:
W20260806 20:16:25.866693 30870 rpcz_store.cc:269] 0806 20:16:24.084262 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.866112 (+1781850us) service_pool.cc:214] Skipping call since client already timed out
0806 20:16:25.866234 (+ 122us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:16:25.834112 1262 leader_election.cc:288] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:24.900002 30706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1625) took 2989 ms (client timeout 1996 ms). Trace:
I20260806 20:16:25.851312 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:25.867666 30706 rpcz_store.cc:269] 0806 20:16:21.910820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.899775 (+2988955us) service_pool.cc:214] Skipping call since client already timed out
0806 20:16:24.899899 (+ 124us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260806 20:16:25.151752 30861 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1504) took 1928 ms (client timeout 2132 ms). Trace:
W20260806 20:16:25.868522 30861 rpcz_store.cc:269] 0806 20:16:23.223060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.558545 (+1335485us) service_pool.cc:227] Handling call
0806 20:16:25.151569 (+593024us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54002fb7d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:25.151688 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":244921728}
I20260806 20:16:25.869196 30712 raft_consensus.cc:2362] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7. Re-sending same reply.
W20260806 20:16:25.824177 1339 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.001s user 0.019s sys 0.029s
W20260806 20:16:24.939462 30866 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1572) took 4778 ms (client timeout 1795 ms). Trace:
I20260806 20:16:24.936661 1208 raft_consensus.cc:482] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:25.869705 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41700 (request call id 411) took 1159 ms. Trace:
W20260806 20:16:25.869801 1339 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.047s user 0.019s sys 0.030s
I20260806 20:16:25.825681 1309 raft_consensus.cc:504] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:25.870064 30712 rpcz_store.cc:276] 0806 20:16:24.709964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.850985 (+1141021us) service_pool.cc:227] Handling call
0806 20:16:25.869653 (+ 18668us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.977973 30709 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:24.762625 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:25.870517 1328 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:25.871075 31417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 559) took 1179 ms. Trace:
I20260806 20:16:25.871356 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:25.820264 30703 rpcz_store.cc:276] 0806 20:16:23.194067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.615300 (+1421233us) service_pool.cc:227] Handling call
0806 20:16:24.737854 (+122554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.858903 1417 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:25.872396 31422 raft_consensus.cc:1226] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260806 20:16:25.854966 30676 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:25.873792 30703 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260806 20:16:25.874382 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1776) took 4531 ms. Trace:
I20260806 20:16:25.874451 1348 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:25.874742 30703 rpcz_store.cc:276] 0806 20:16:21.343216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.873221 (+4530005us) service_pool.cc:227] Handling call
0806 20:16:25.873578 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:25.874138 (+ 560us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.874330 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.875567 30676 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:25.837842 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41700 (request call id 412) took 1095 ms. Trace:
I20260806 20:16:25.876371 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:25.872856 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:25.876976 30708 rpcz_store.cc:276] 0806 20:16:24.742463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.769857 (+ 27394us) service_pool.cc:227] Handling call
0806 20:16:25.837775 (+1067918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.868095 31414 rpcz_store.cc:276] 0806 20:16:24.724477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.724605 (+ 128us) service_pool.cc:227] Handling call
0806 20:16:24.793442 (+ 68837us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:25.821721 (+1028279us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.821941 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.877189 30863 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 9: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
W20260806 20:16:25.150089 30867 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 246) took 4973 ms (client timeout 2325 ms). Trace:
I20260806 20:16:25.877677 30712 raft_consensus.cc:2362] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 10. Re-sending same reply.
I20260806 20:16:25.876286 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1777) took 4532 ms. Trace:
I20260806 20:16:25.154243 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:25.878118 30706 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 9: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
W20260806 20:16:25.879004 30863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1582) took 1906 ms (client timeout 1960 ms). Trace:
W20260806 20:16:25.062245 31389 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.003s sys 0.007s
W20260806 20:16:25.035945 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1481) took 2728 ms (client timeout 1719 ms). Trace:
I20260806 20:16:24.932449 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:25.845243 1320 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:25.889807 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 63216.4us
I20260806 20:16:24.938663 1361 raft_consensus.cc:3032] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:25.840818 1382 leader_election.cc:288] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:25.846321 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:25.775177 1415 log.cc:924] Time spent T 7e6251f4f1aa48ca869b81256b9fd89a P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.580s user 0.001s sys 0.000s
I20260806 20:16:24.883498 1407 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:24.869473 986 consensus_meta.cc:287] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043: Time spent flushing consensus metadata: real 1.136s user 0.007s sys 0.004s
I20260806 20:16:25.007550 1337 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:24.895306 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:25.912247 31420 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:25.867089 1076 raft_consensus.cc:3037] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:24.769661 1405 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:25.871413 31417 rpcz_store.cc:276] 0806 20:16:24.691500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.721906 (+ 30406us) service_pool.cc:227] Handling call
0806 20:16:25.871016 (+1149110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.870060 1208 raft_consensus.cc:504] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:25.070555 30792 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1525) took 1368 ms (client timeout 1658 ms). Trace:
I20260806 20:16:25.872721 878 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:25.879973 30703 rpcz_store.cc:276] 0806 20:16:21.343523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.875537 (+4532014us) service_pool.cc:227] Handling call
0806 20:16:25.875848 (+ 311us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:25.876003 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:25.876025 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:25.876067 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:25.876095 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.876140 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:25.876240 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:24.940979 1390 raft_consensus.cc:2726] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:24.819275 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:25.872396 30709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1798) took 2553 ms (client timeout 3142 ms). Trace:
W20260806 20:16:25.921636 30709 rpcz_store.cc:269] 0806 20:16:23.319375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.690192 (+1370817us) service_pool.cc:227] Handling call
0806 20:16:25.872337 (+1182145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.844094 31418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 558) took 1152 ms. Trace:
W20260806 20:16:25.870150 30866 rpcz_store.cc:269] 0806 20:16:20.161231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.173615 (+ 12384us) service_pool.cc:227] Handling call
0806 20:16:24.939220 (+4765605us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x7b540007e754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:24.939352 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97882880}
W20260806 20:16:25.880982 30869 rpcz_store.cc:269] 0806 20:16:22.281144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:22.281281 (+ 137us) service_pool.cc:227] Handling call
0806 20:16:25.009517 (+2728236us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b540014c3d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:25.009655 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1355314048}
I20260806 20:16:25.867208 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:25.884156 30868 rpcz_store.cc:269] 0806 20:16:24.026131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.850857 (+1824726us) service_pool.cc:214] Skipping call since client already timed out
0806 20:16:25.851010 (+ 153us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:16:25.947674 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
W20260806 20:16:25.884835 31389 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.367s user 0.018s sys 0.018s
I20260806 20:16:25.886968 31388 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:25.887236 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 401) took 4531 ms. Trace:
I20260806 20:16:25.948824 30757 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:25.878504 30784 rpcz_store.cc:276] 0806 20:16:23.682657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.698547 (+1015890us) service_pool.cc:227] Handling call
0806 20:16:24.725527 (+ 26980us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:24.725705 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:24.725728 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:24.725767 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:24.725803 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:24.725851 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:24.725947 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:25.880712 30863 rpcz_store.cc:269] 0806 20:16:23.972243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.865220 (+1892977us) service_pool.cc:227] Handling call
0806 20:16:25.878952 (+ 13732us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.948889 30706 rpcz_store.cc:276] 0806 20:16:21.356169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.886418 (+4530249us) service_pool.cc:227] Handling call
0806 20:16:25.886731 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:25.886912 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:25.886935 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:25.886980 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:25.887019 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.887074 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:25.887172 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.858461 30786 rpcz_store.cc:276] 0806 20:16:23.697474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.852914 (+2155440us) service_pool.cc:227] Handling call
0806 20:16:25.853337 (+ 423us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:25.853568 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:25.853594 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:25.853633 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:25.853714 (+ 81us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.853769 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:25.853879 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.949611 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1784) took 3739 ms. Trace:
I20260806 20:16:24.921253 30754 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:25.890643 30835 leader_election.cc:302] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:25.950014 30713 rpcz_store.cc:276] 0806 20:16:22.209909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.866423 (+3656514us) service_pool.cc:227] Handling call
0806 20:16:25.949548 (+ 83125us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:25.888072 30867 rpcz_store.cc:269] 0806 20:16:20.176838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.184261 (+ 7423us) service_pool.cc:227] Handling call
0806 20:16:25.149797 (+4965536us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b54000b0754. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:25.149985 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2102186752}
W20260806 20:16:25.892702 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 57977.3us
I20260806 20:16:25.871766 1309 leader_election.cc:288] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 11 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:25.882962 1413 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.001s sys 0.031s
I20260806 20:16:25.905486 30676 leader_election.cc:302] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:25.227640 1009 raft_consensus.cc:3032] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:25.903901 30864 raft_consensus.cc:2454] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
W20260806 20:16:25.953428 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 66546.8us
W20260806 20:16:25.954085 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 63527.7us
W20260806 20:16:25.954247 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 48875.2us
I20260806 20:16:25.913133 31420 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 565) took 1030 ms. Trace:
W20260806 20:16:25.955883 30864 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1584) took 1983 ms (client timeout 1802 ms). Trace:
W20260806 20:16:25.959807 30864 rpcz_store.cc:269] 0806 20:16:23.972850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.664353 (+1691503us) service_pool.cc:227] Handling call
0806 20:16:25.955814 (+291461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.957628 31420 rpcz_store.cc:276] 0806 20:16:24.883132 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.883321 (+ 189us) service_pool.cc:227] Handling call
0806 20:16:25.913060 (+1029739us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.954754 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:25.965981 30706 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 3.
I20260806 20:16:25.962714 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:25.967543 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:25.967840 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:25.968523 30713 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:25.971380 30784 raft_consensus.cc:1261] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260806 20:16:25.971799 30784 pending_rounds.cc:76] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Aborting all ops after (but not including) 7
I20260806 20:16:25.972015 30784 pending_rounds.cc:97] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260806 20:16:25.972262 30784 raft_consensus.cc:2866] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:25.972956 30784 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 347) took 1456 ms. Trace:
I20260806 20:16:25.974573 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:25.975279 30864 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:25.976130 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:25.973295 30784 rpcz_store.cc:276] 0806 20:16:24.516515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.954815 (+1438300us) service_pool.cc:227] Handling call
0806 20:16:25.970999 (+ 16184us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:25.972690 (+ 1691us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.972895 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.977170 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:25.846378 1353 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:25.890043 1408 raft_consensus.cc:2726] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:25.946496 31418 rpcz_store.cc:276] 0806 20:16:24.691177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.691821 (+ 644us) service_pool.cc:227] Handling call
0806 20:16:25.844029 (+1152208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.962795 1338 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:25.981158 1338 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:25.904881 1292 raft_consensus.cc:482] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:25.982769 30789 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53732 (request call id 343) took 2328 ms. Trace:
I20260806 20:16:25.983157 30789 rpcz_store.cc:276] 0806 20:16:23.654817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.598336 (+943519us) service_pool.cc:227] Handling call
0806 20:16:24.950559 (+352223us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:25.862228 (+911669us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.982587 (+120359us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54001423d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:25.982713 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":809400704}
I20260806 20:16:25.983891 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:25.984818 30861 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1589) took 1593 ms (client timeout 1658 ms). Trace:
I20260806 20:16:25.962468 1239 raft_consensus.cc:2781] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260806 20:16:25.985185 30861 rpcz_store.cc:269] 0806 20:16:24.391316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.869332 (+1478016us) service_pool.cc:227] Handling call
0806 20:16:25.983722 (+114390us) spinlock_profiling.cc:243] Waited 7.03 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:16:25.984755 (+ 1033us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16164224}
I20260806 20:16:25.962743 1348 raft_consensus.cc:2781] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:25.985808 1239 raft_consensus.cc:482] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:25.935174 1374 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:25.903426 1361 raft_consensus.cc:729] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:25.962708 1380 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260806 20:16:25.963228 1309 raft_consensus.cc:2726] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:25.943252 1405 raft_consensus.cc:686] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:25.983181 1292 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:25.955622 1009 raft_consensus.cc:729] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:25.988945 1405 consensus_queue.cc:237] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:25.990207 1009 consensus_queue.cc:260] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:25.990716 1292 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:25.982677 1204 leader_election.cc:288] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:25.986300 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:25.992504 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1511) took 1012 ms. Trace:
I20260806 20:16:25.992795 30861 rpcz_store.cc:276] 0806 20:16:24.980174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.986001 (+1005827us) service_pool.cc:227] Handling call
0806 20:16:25.992438 (+ 6437us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:25.980836 30705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1624) took 4073 ms (client timeout 2320 ms). Trace:
W20260806 20:16:25.993479 30705 rpcz_store.cc:269] 0806 20:16:21.907832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.980610 (+4072778us) service_pool.cc:214] Skipping call since client already timed out
0806 20:16:25.980762 (+ 152us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 20:16:25.993822 1338 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:25.994176 30861 raft_consensus.cc:1261] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 20:16:25.994442 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:25.994772 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1483) took 3680 ms. Trace:
I20260806 20:16:25.995069 30705 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Advancing to term 11
I20260806 20:16:25.995121 30861 rpcz_store.cc:276] 0806 20:16:22.314245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.993574 (+3679329us) service_pool.cc:227] Handling call
0806 20:16:25.993878 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:25.994548 (+ 670us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.994727 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.992906 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:25.986305 1239 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Advancing to term 11
I20260806 20:16:25.997311 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:25.997793 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1491) took 3429 ms. Trace:
I20260806 20:16:25.998202 30867 raft_consensus.cc:2396] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 20:16:25.998700 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1510) took 1018 ms. Trace:
I20260806 20:16:25.998968 30867 rpcz_store.cc:276] 0806 20:16:24.979879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.970805 (+990926us) service_pool.cc:227] Handling call
0806 20:16:25.998634 (+ 27829us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.991271 1009 raft_consensus.cc:3037] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:26.003489 31388 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:26.006492 1041 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 20:16:26.007037 30705 raft_consensus.cc:2454] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 11.
I20260806 20:16:26.007977 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.008749 30705 raft_consensus.cc:2379] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 9: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:26.008783 30788 raft_consensus.cc:1261] T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 20:16:26.012630 30788 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 1991) took 4227 ms. Trace:
I20260806 20:16:25.927456 1208 leader_election.cc:288] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:25.929509 31419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49038 (request call id 567) took 1006 ms. Trace:
I20260806 20:16:26.013036 30788 rpcz_store.cc:276] 0806 20:16:21.784831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.008207 (+4223376us) service_pool.cc:227] Handling call
0806 20:16:26.008486 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.012380 (+ 3894us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.012573 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:25.924141 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c09420eb1e4f3692c165f76cd1a7d4" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:25.931291 30787 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33226 (request call id 2013) took 2751 ms (client timeout 3142 ms). Trace:
I20260806 20:16:25.913040 1407 raft_consensus.cc:504] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.023137 1407 leader_election.cc:288] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:25.903926 1417 consensus_queue.cc:1046] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260806 20:16:25.905936 1320 consensus_queue.cc:1046] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260806 20:16:25.913467 986 raft_consensus.cc:504] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.025375 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.227969 30703 raft_consensus.cc:2379] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:26.229872 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.231546 30703 raft_consensus.cc:2396] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 20:16:26.232791 30703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.012603 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:25.968434 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b5901783b8423c9e494c131b0a64e9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:25.922505 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec60d6f5b6c486b80781ad3cb85a06b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.235400 30705 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:26.235948 30709 raft_consensus.cc:2379] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 8: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:26.236720 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.235836 30706 raft_consensus.cc:2396] T 47b5901783b8423c9e494c131b0a64e9 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260806 20:16:26.237612 30705 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:26.238438 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.238873 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.239862 30706 raft_consensus.cc:2379] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 10: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:26.237227 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.243328 30709 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:16:26.244544 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:26.245837 30709 raft_consensus.cc:2379] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 9: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:26.247040 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.248486 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:26.249320 30709 raft_consensus.cc:2362] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7. Re-sending same reply.
I20260806 20:16:26.241226 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:25.951174 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 1.02996e+06us
I20260806 20:16:26.239892 30705 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
W20260806 20:16:26.037438 30756 leader_election.cc:334] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Timed out: RequestConsensusVote RPC to 127.28.186.193:42937 timed out after 1.943s (SENT)
I20260806 20:16:26.253149 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.254287 30705 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 9.
I20260806 20:16:26.252976 30756 leader_election.cc:302] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:26.257087 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 221443us
I20260806 20:16:26.258805 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.261245 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec226f72b38a49529b88339beb36af46" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.262017 30706 raft_consensus.cc:2454] T ec226f72b38a49529b88339beb36af46 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
W20260806 20:16:26.259611 30756 leader_election.cc:334] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Timed out: RequestConsensusVote RPC to 127.28.186.193:42937 timed out after 2.341s (SENT)
W20260806 20:16:25.916352 30792 rpcz_store.cc:269] 0806 20:16:23.702393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.818275 (+115882us) service_pool.cc:227] Handling call
0806 20:16:25.070332 (+1252057us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b5400149bd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:25.070474 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1401032576}
I20260806 20:16:26.259909 30754 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:26.015352 30787 rpcz_store.cc:269] 0806 20:16:23.179860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.179998 (+ 138us) service_pool.cc:227] Handling call
0806 20:16:25.931108 (+2751110us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54003a07d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:25.931227 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2820641024}
I20260806 20:16:26.266453 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616aae135c70498da00a9e8a1a61b1cd" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.267544 30713 raft_consensus.cc:2454] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 9.
I20260806 20:16:26.268394 30787 raft_consensus.cc:1226] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 2. Current term is 3. Ops: []
I20260806 20:16:26.039085 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.272421 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.273375 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:26.167222 30835 leader_election.cc:302] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:25.998169 30861 rpcz_store.cc:276] 0806 20:16:22.568257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.996067 (+3427810us) service_pool.cc:227] Handling call
0806 20:16:25.996369 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:25.996543 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:25.996569 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:25.996605 (+ 36us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:25.996634 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.997628 (+ 994us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:25.997742 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:26.283186 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 115888us
I20260806 20:16:25.952364 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:26.297529 30866 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1583) took 2324 ms (client timeout 2113 ms). Trace:
W20260806 20:16:26.298015 30866 rpcz_store.cc:269] 0806 20:16:23.972494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.946892 (+1974398us) service_pool.cc:227] Handling call
0806 20:16:26.297174 (+350282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.301398 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.306107 30706 raft_consensus.cc:3037] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Advancing to term 7
I20260806 20:16:26.307102 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.304265 878 raft_consensus.cc:2781] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:26.308698 878 raft_consensus.cc:482] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:26.309175 878 raft_consensus.cc:3037] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:26.315177 31421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:26.029418 30867 raft_consensus.cc:1226] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260806 20:16:25.948390 30868 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1587) took 1870 ms (client timeout 2100 ms). Trace:
W20260806 20:16:26.320145 30868 rpcz_store.cc:269] 0806 20:16:24.078309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.947402 (+1869093us) service_pool.cc:227] Handling call
0806 20:16:25.948336 (+ 934us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.322633 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fd0995e74a449090ea2b00062f625e" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.326025 1019 raft_consensus.cc:2726] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 20:16:25.952517 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:26.328459 30869 raft_consensus.cc:2379] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
W20260806 20:16:26.329140 30869 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1506) took 3076 ms (client timeout 2906 ms). Trace:
W20260806 20:16:26.329638 30869 rpcz_store.cc:269] 0806 20:16:23.252751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.952279 (+2699528us) service_pool.cc:227] Handling call
0806 20:16:26.329018 (+376739us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.103143 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1513) took 1122 ms. Trace:
I20260806 20:16:26.333153 30864 rpcz_store.cc:276] 0806 20:16:24.980672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.976948 (+996276us) service_pool.cc:227] Handling call
0806 20:16:26.103055 (+126107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.333820 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.333357 30754 leader_election.cc:302] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:26.170957 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53758bb873349479f86e0abf34c738b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:25.922783 31415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49038 (request call id 563) took 1196 ms. Trace:
I20260806 20:16:26.335868 31415 rpcz_store.cc:276] 0806 20:16:24.726714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.726838 (+ 124us) service_pool.cc:227] Handling call
0806 20:16:25.922708 (+1195870us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.170329 31417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c52f63d3e8412bbd0e51fe5b81568b" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:26.174039 31418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.013967 31419 rpcz_store.cc:276] 0806 20:16:24.922768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.923007 (+ 239us) service_pool.cc:227] Handling call
0806 20:16:24.931991 (+ 8984us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:25.822161 (+890170us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:25.822202 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:25.822239 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:25.822268 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:25.929236 (+106968us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x7b54005d0554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:25.929290 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:25.929429 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2046788864}
I20260806 20:16:26.263206 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.362771 1301 raft_consensus.cc:482] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:26.371656 30709 raft_consensus.cc:2454] T 7cb9334179bc40b1b3868fc4773af362 P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:26.371901 1301 raft_consensus.cc:504] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.372834 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.177971 31422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48918 (request call id 454) took 4431 ms. Trace:
I20260806 20:16:26.168556 31423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.026789 986 leader_election.cc:288] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:26.007253 1041 raft_consensus.cc:686] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:25.836959 1120 raft_consensus.cc:504] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.388962 1345 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:26.389946 1041 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.391193 1120 leader_election.cc:288] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:26.108174 1226 raft_consensus.cc:482] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 20:16:26.391697 30790 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1514) took 2712 ms (client timeout 1548 ms). Trace:
W20260806 20:16:26.392196 30790 rpcz_store.cc:269] 0806 20:16:23.678968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.689580 (+ 10612us) service_pool.cc:227] Handling call
0806 20:16:26.391299 (+2701719us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b540026f554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:26.391457 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1574326528}
I20260806 20:16:26.392555 1226 raft_consensus.cc:504] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.394039 31421 raft_consensus.cc:2362] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7. Re-sending same reply.
I20260806 20:16:26.394477 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 250) took 6216 ms. Trace:
I20260806 20:16:25.945902 1288 raft_consensus.cc:482] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:26.394925 30871 rpcz_store.cc:276] 0806 20:16:20.178409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.184274 (+ 5865us) service_pool.cc:227] Handling call
0806 20:16:24.891981 (+4707707us) spinlock_profiling.cc:243] Waited 4.45 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565c52b0 00007f71565b07f9 00007f71565b02a2 00007f714fca2684
0806 20:16:24.892232 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:25.856402 (+964170us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.393979 (+537577us) spinlock_profiling.cc:243] Waited 947 ms on lock 0x7b54002bb1d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:26.394268 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2188840576}
I20260806 20:16:26.396797 30871 raft_consensus.cc:1226] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:26.102233 1320 consensus_queue.cc:1035] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260806 20:16:26.393929 987 raft_consensus.cc:482] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:26.406783 1288 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:26.407691 30783 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33236 (request call id 1518) took 2725 ms (client timeout 1602 ms). Trace:
I20260806 20:16:26.189894 1322 leader_election.cc:288] T b32e68e22f574e679ad1e327caa342f6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:26.408123 30783 rpcz_store.cc:269] 0806 20:16:23.681758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.698809 (+ 17051us) service_pool.cc:227] Handling call
0806 20:16:26.407286 (+2708477us) spinlock_profiling.cc:243] Waited 2.43 s on lock 0x7b54000573d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:26.407494 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5584002816}
I20260806 20:16:25.986181 1348 raft_consensus.cc:686] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:26.178035 1380 consensus_queue.cc:1035] T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
W20260806 20:16:26.410774 30710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59116 (request call id 1621) took 4684 ms (client timeout 2162 ms). Trace:
I20260806 20:16:26.411031 1348 consensus_queue.cc:237] T b32e68e22f574e679ad1e327caa342f6 P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:25.953092 1390 raft_consensus.cc:2726] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:26.169407 1407 consensus_queue.cc:1035] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260806 20:16:26.412655 1226 leader_election.cc:288] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:26.414896 30789 raft_consensus.cc:2454] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 5.
I20260806 20:16:26.415870 30708 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:25.938126 1398 leader_election.cc:288] T b060012888ff4d76b254b43d1f1959bb P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:25.953884 1413 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.952s user 0.001s sys 0.032s
I20260806 20:16:26.024231 1417 raft_consensus.cc:3032] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:26.417829 1417 raft_consensus.cc:729] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:26.419303 1417 consensus_queue.cc:260] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.191349 1411 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:26.273741 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.422175 30713 raft_consensus.cc:2454] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:26.423110 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.424389 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:26.293781 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 372834us
I20260806 20:16:26.425379 878 raft_consensus.cc:504] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:26.427176 30710 rpcz_store.cc:269] 0806 20:16:21.726134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:22.108625 (+382491us) service_pool.cc:227] Handling call
0806 20:16:26.410259 (+4301634us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b5400270954. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:26.410536 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":930802560}
I20260806 20:16:26.428483 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.428582 30834 leader_election.cc:302] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:26.294580 30861 pending_rounds.cc:76] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 1
I20260806 20:16:26.429472 30861 pending_rounds.cc:97] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 20:16:26.429733 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1778) took 5076 ms. Trace:
I20260806 20:16:26.429821 30861 raft_consensus.cc:2866] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:26.430191 30708 rpcz_store.cc:276] 0806 20:16:21.352903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.878723 (+4525820us) service_pool.cc:227] Handling call
0806 20:16:26.415352 (+536629us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.416456 (+ 1104us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.417404 (+ 948us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400270950. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:26.429637 (+ 12233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1170436480}
I20260806 20:16:26.317402 1305 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:26.319242 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1492) took 3750 ms. Trace:
W20260806 20:16:25.891552 30865 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60894 (request call id 1496) took 3282 ms (client timeout 2162 ms). Trace:
I20260806 20:16:26.319336 30712 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1780) took 4965 ms. Trace:
I20260806 20:16:26.431169 1305 raft_consensus.cc:482] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:26.335536 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.358881 1019 raft_consensus.cc:2726] T 01fd0995e74a449090ea2b00062f625e P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 20:16:26.325001 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.366937 1338 raft_consensus.cc:504] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.432334 31420 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:26.374843 30709 raft_consensus.cc:2379] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:26.373512 30707 raft_consensus.cc:2454] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
I20260806 20:16:26.434580 30713 raft_consensus.cc:2379] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:26.435947 31416 raft_consensus.cc:2454] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:26.431224 30867 rpcz_store.cc:276] 0806 20:16:22.568504 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.999527 (+3431023us) service_pool.cc:227] Handling call
0806 20:16:25.999848 (+ 321us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:26.316661 (+316813us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.317543 (+ 882us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b54002fb2d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:26.319000 (+ 1457us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67388672}
I20260806 20:16:26.431569 30712 rpcz_store.cc:276] 0806 20:16:21.353531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.880193 (+4526662us) service_pool.cc:227] Handling call
0806 20:16:25.904978 (+ 24785us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:25.905303 (+ 325us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:25.905331 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:25.912039 (+ 6708us) log.cc:842] Serialized 26 byte log entry
0806 20:16:26.237483 (+325444us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:26.237634 (+ 151us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.237769 (+ 135us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:26.318077 (+ 80308us) raft_consensus.cc:1671] finished
0806 20:16:26.318094 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.319115 (+ 1021us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":591,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":7148,"thread_start_us":290711,"threads_started":1,"wal-append.queue_time_us":391015}
I20260806 20:16:26.409198 30863 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 6.
I20260806 20:16:26.438848 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.321869 1239 raft_consensus.cc:504] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:26.431592 30865 rpcz_store.cc:269] 0806 20:16:22.609412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:23.690055 (+1080643us) service_pool.cc:227] Handling call
0806 20:16:25.154057 (+1464002us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:16:25.891480 (+737423us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2856704}
I20260806 20:16:26.080171 1309 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:26.394950 30703 raft_consensus.cc:2379] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 9: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
W20260806 20:16:26.402204 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 104913us
I20260806 20:16:26.446070 30712 raft_consensus.cc:2454] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:26.407017 987 raft_consensus.cc:504] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.411897 30785 raft_consensus.cc:1226] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:25.987094 1361 consensus_queue.cc:260] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.453125 1239 leader_election.cc:288] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 11 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:26.285778 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.424894 1346 leader_election.cc:288] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:26.453881 30784 raft_consensus.cc:1226] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 11 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260806 20:16:26.431770 1305 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:26.382018 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:26.375315 31422 rpcz_store.cc:276] 0806 20:16:21.746198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:21.746356 (+ 158us) service_pool.cc:227] Handling call
0806 20:16:21.763906 (+ 17550us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:25.872834 (+4108928us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.177342 (+304508us) spinlock_profiling.cc:243] Waited 4.11 s on lock 0x7b54004c2cd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:26.177578 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9448129920}
I20260806 20:16:26.453835 1361 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:26.435552 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.441792 30706 raft_consensus.cc:2454] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:26.437340 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.435966 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 260) took 3208 ms. Trace:
I20260806 20:16:26.458284 30871 rpcz_store.cc:276] 0806 20:16:23.196368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.396012 (+3199644us) service_pool.cc:227] Handling call
0806 20:16:26.396509 (+ 497us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.404736 (+ 8227us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.404962 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.433880 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:26.433879 1338 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:26.443953 1288 leader_election.cc:288] T 0f3393f94b2649378cb19b57fecfb21e P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:26.448786 1384 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:26.429824 878 leader_election.cc:288] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:26.457912 30710 raft_consensus.cc:2454] T e1485b55849242feae0e8b2b3b28307f P 42cabca48db7403b8523a3382a046043 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
W20260806 20:16:26.455412 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 261757us
I20260806 20:16:26.460201 30709 raft_consensus.cc:2379] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:26.462045 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:26.459992 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f7b0d94b4e4047a32e7fbdde6c35d5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.480537 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:26.482153 30754 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:26.468700 1417 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:26.484624 30705 raft_consensus.cc:2454] T e0c1793b412c4608b1e004525a6dd887 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:26.489194 30713 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:26.488060 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1577) took 2512 ms. Trace:
I20260806 20:16:26.498476 30871 raft_consensus.cc:1226] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:26.499382 30871 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60902 (request call id 1581) took 2527 ms. Trace:
I20260806 20:16:26.499889 961 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 3 FOLLOWER]: Advancing to term 4
I20260806 20:16:26.500728 30708 raft_consensus.cc:2396] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260806 20:16:26.501802 1309 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.496483 30865 rpcz_store.cc:276] 0806 20:16:23.970984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.448708 (+2477724us) service_pool.cc:227] Handling call
0806 20:16:26.455147 (+ 6439us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.483314 (+ 28167us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.483346 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.483395 (+ 49us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:26.483431 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.483488 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.483614 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:26.503034 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 74583.8us
I20260806 20:16:26.503979 30865 raft_consensus.cc:1226] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:26.504034 30834 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:26.504526 1309 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:26.504778 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 263) took 1753 ms. Trace:
I20260806 20:16:26.505868 30709 raft_consensus.cc:1261] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer 58f2f6256ee14cb38a1dbcbf5bb590a0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:16:26.506631 30709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 402) took 5138 ms. Trace:
I20260806 20:16:26.506966 30709 rpcz_store.cc:276] 0806 20:16:21.368607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.482566 (+5113959us) service_pool.cc:227] Handling call
0806 20:16:26.505357 (+ 22791us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.506344 (+ 987us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.506560 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.505760 30865 rpcz_store.cc:276] 0806 20:16:24.751206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.503263 (+1752057us) service_pool.cc:227] Handling call
0806 20:16:26.503675 (+ 412us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.504526 (+ 851us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.504710 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.509948 30712 raft_consensus.cc:3037] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:26.511435 987 leader_election.cc:288] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:26.511919 1337 consensus_queue.cc:1035] T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 20:16:26.513069 1293 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:26.502348 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1783) took 5035 ms. Trace:
I20260806 20:16:26.499825 30871 rpcz_store.cc:276] 0806 20:16:23.971988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.481173 (+2509185us) service_pool.cc:227] Handling call
0806 20:16:26.498124 (+ 16951us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.499054 (+ 930us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.499316 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:26.494645 30870 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60902 (request call id 1585) took 2515 ms (client timeout 1968 ms). Trace:
I20260806 20:16:26.520272 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1607) took 6098 ms. Trace:
I20260806 20:16:26.522768 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1781) took 5165 ms. Trace:
I20260806 20:16:26.499284 1374 raft_consensus.cc:2781] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:26.511382 30835 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:26.528585 30710 rpcz_store.cc:276] 0806 20:16:21.365073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.485972 (+5120899us) service_pool.cc:227] Handling call
0806 20:16:26.522012 (+ 36040us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.522377 (+ 365us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.522405 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.522462 (+ 57us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:26.522504 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.522554 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.522684 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:26.526752 30870 rpcz_store.cc:269] 0806 20:16:23.973212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.867278 (+1894066us) service_pool.cc:227] Handling call
0806 20:16:26.378491 (+511213us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5800275798. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e71b851 00007f714e71b702 00007f71575ffb09 00007f715653b730 00007f715660179b 00007f7156601821 00007f71565b11b0 00007f71565b0a94 00007f714fca4c04 00007f714fca4b92
0806 20:16:26.488524 (+110033us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34054016}
I20260806 20:16:26.525558 30706 rpcz_store.cc:276] 0806 20:16:21.466636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.497169 (+5030533us) service_pool.cc:227] Handling call
0806 20:16:26.501634 (+ 4465us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.501969 (+ 335us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.501991 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.502047 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:26.502094 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.502159 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.502271 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.509596 30676 leader_election.cc:302] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:26.527917 30711 rpcz_store.cc:276] 0806 20:16:20.421633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:20.421743 (+ 110us) service_pool.cc:227] Handling call
0806 20:16:20.422031 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:21.981825 (+1559794us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.519698 (+4537873us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540039f8d4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ff51 00007f715166e99e 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86
0806 20:16:26.520063 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3416926592}
I20260806 20:16:26.529649 1338 consensus_queue.cc:1035] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260806 20:16:26.524855 30754 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:26.539978 1311 raft_consensus.cc:2726] T f92e9dc9db864f0b8e238a39be31b12e P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:26.541879 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1782) took 5075 ms. Trace:
I20260806 20:16:26.542387 30705 rpcz_store.cc:276] 0806 20:16:21.466339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.518781 (+5052442us) service_pool.cc:227] Handling call
0806 20:16:26.541024 (+ 22243us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.541508 (+ 484us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.541552 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.541595 (+ 43us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:26.541638 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.541692 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.541804 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.543684 30865 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:60882 (request call id 807) took 2609 ms. Trace:
I20260806 20:16:26.544137 30865 rpcz_store.cc:276] 0806 20:16:23.969965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.536276 (+2566311us) service_pool.cc:227] Handling call
0806 20:16:26.543586 (+ 7310us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.545971 30783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "942abb68076e47a0a24672bb92649892" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.547844 1407 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:26.548373 1407 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:26.548815 1407 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:26.549788 30864 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 259) took 3406 ms. Trace:
I20260806 20:16:26.549541 1384 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.550244 30864 rpcz_store.cc:276] 0806 20:16:23.143136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.334178 (+3191042us) service_pool.cc:227] Handling call
0806 20:16:26.334789 (+ 611us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:26.341255 (+ 6466us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.341290 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:26.381257 (+ 39967us) log.cc:842] Serialized 26 byte log entry
0806 20:16:26.414974 (+ 33717us) spinlock_profiling.cc:243] Waited 2.41 ms on lock 0x7b480027c0d8. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71565167f1 00007f715650d8e7 00007f715650d77a 00007f715660195f 00007f7156601627 00007f7156615b2c 00007f7156615ad8 00007f7156615a58 00007f7156615a13 00007f7156614cbc 00007f7155596be6
0806 20:16:26.460247 (+ 45273us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:26.460286 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.460413 (+ 127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:26.549427 (+ 89014us) raft_consensus.cc:1671] finished
0806 20:16:26.549490 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.549700 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5715,"raft.queue_time_us":6878,"raft.run_cpu_time_us":3185,"raft.run_wall_time_us":72293,"spinlock_wait_cycles":5542784,"thread_start_us":71735,"threads_started":1,"wal-append.queue_time_us":158742}
I20260806 20:16:26.551723 30676 leader_election.cc:302] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:26.552080 1384 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:26.552887 30869 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1500) took 3468 ms. Trace:
I20260806 20:16:26.553833 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.553260 30869 rpcz_store.cc:276] 0806 20:16:23.085126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.330605 (+3245479us) service_pool.cc:227] Handling call
0806 20:16:26.342304 (+ 11699us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.366534 (+ 24230us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 20:16:26.428248 (+ 61714us) log.cc:842] Serialized 14 byte log entry
0806 20:16:26.552541 (+124293us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.552594 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:26.552646 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.552714 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.552827 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36864,"prepare.queue_time_us":121392,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"thread_start_us":147855,"threads_started":2}
I20260806 20:16:26.555548 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f3393f94b2649378cb19b57fecfb21e" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.556432 31422 raft_consensus.cc:2379] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
W20260806 20:16:26.558526 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 47264.7us
W20260806 20:16:26.558864 30863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45134 (request call id 261) took 2664 ms (client timeout 2314 ms). Trace:
W20260806 20:16:26.559305 30863 rpcz_store.cc:269] 0806 20:16:23.894321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:25.949878 (+2055557us) service_pool.cc:227] Handling call
0806 20:16:26.558625 (+608747us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b5400358254. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:26.558795 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1015196928}
W20260806 20:16:26.539826 30756 leader_election.cc:334] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937): Timed out: RequestConsensusVote RPC to 127.28.186.193:42937 timed out after 1.517s (SENT)
I20260806 20:16:26.559365 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.544544 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1628) took 4366 ms. Trace:
I20260806 20:16:26.563522 30705 rpcz_store.cc:276] 0806 20:16:22.177938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.543432 (+4365494us) service_pool.cc:227] Handling call
0806 20:16:26.543857 (+ 425us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.544186 (+ 329us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.544219 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.544274 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:26.544312 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.544370 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.544476 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.570761 30756 leader_election.cc:302] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:26.573351 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 76768.5us
I20260806 20:16:26.570853 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d6e84eb2874b2b9dca3387b3d98eff" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.574898 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:26.569160 30868 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1498) took 3868 ms. Trace:
I20260806 20:16:26.589346 1286 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:26.589965 1286 raft_consensus.cc:482] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:26.590353 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.591128 30705 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:26.590408 1286 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:26.591534 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 406) took 4686 ms. Trace:
I20260806 20:16:26.592191 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.592844 30705 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 3.
I20260806 20:16:26.594993 30834 leader_election.cc:302] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:26.596025 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 44572.9us
I20260806 20:16:26.596637 30834 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:26.597545 1309 raft_consensus.cc:2726] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 20:16:26.597812 987 raft_consensus.cc:2781] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:26.598816 987 raft_consensus.cc:482] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:26.599171 987 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:26.599869 30791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33226 (request call id 2012) took 3429 ms. Trace:
I20260806 20:16:26.600891 30791 rpcz_store.cc:276] 0806 20:16:23.170323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.027446 (+2857123us) service_pool.cc:227] Handling call
0806 20:16:26.027827 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.028033 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 20:16:26.384352 (+356319us) log.cc:842] Serialized 14 byte log entry
0806 20:16:26.598889 (+214537us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.598978 (+ 89us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:26.599068 (+ 90us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.599549 (+ 481us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.599694 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3104,"prepare.queue_time_us":285885,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":77,"raft.queue_time_us":17454,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":116,"thread_start_us":468871,"threads_started":2}
W20260806 20:16:26.607024 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 672b273ef36e4c6eb6873cc61471cb53, TS d4181d4d907f477f9feddc44ba3b9207 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:26.609826 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 672b273ef36e4c6eb6873cc61471cb53, TS d4181d4d907f477f9feddc44ba3b9207 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.606936 1286 raft_consensus.cc:504] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.601729 1384 consensus_queue.cc:1046] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260806 20:16:26.607479 1348 raft_consensus.cc:2781] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:26.591987 30711 rpcz_store.cc:276] 0806 20:16:21.905225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.561274 (+4656049us) service_pool.cc:227] Handling call
0806 20:16:26.590967 (+ 29693us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.591196 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.591224 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.591283 (+ 59us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:26.591309 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.591366 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.591471 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.578722 31389 leader_election.cc:302] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:26.618289 30754 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:26.620751 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.621755 30711 raft_consensus.cc:2379] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 6: Already voted for candidate 42cabca48db7403b8523a3382a046043 in this term.
I20260806 20:16:26.597290 30676 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:26.626106 1293 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:26.626757 1293 raft_consensus.cc:686] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:26.627795 1293 consensus_queue.cc:237] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.601567 1337 consensus_queue.cc:1046] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260806 20:16:26.631697 1337 raft_consensus.cc:3032] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:26.632144 1337 raft_consensus.cc:729] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:26.635934 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59116 (request call id 1622) took 4909 ms. Trace:
I20260806 20:16:26.636382 30710 rpcz_store.cc:276] 0806 20:16:21.726407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.539578 (+4813171us) service_pool.cc:227] Handling call
0806 20:16:26.623709 (+ 84131us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.635518 (+ 11809us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.635552 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.635613 (+ 61us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:26.635660 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.635720 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.635845 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.637429 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.638188 1398 raft_consensus.cc:2781] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:26.638665 1398 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:26.639045 1398 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:26.644762 1443 log.cc:924] Time spent T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
W20260806 20:16:26.645533 1438 log.cc:924] Time spent T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260806 20:16:26.646620 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 405) took 4815 ms. Trace:
I20260806 20:16:26.647058 30706 rpcz_store.cc:276] 0806 20:16:21.831483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.537332 (+4705849us) service_pool.cc:227] Handling call
0806 20:16:26.634122 (+ 96790us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.646253 (+ 12131us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.646284 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.646330 (+ 46us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:26.646362 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.646423 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.646559 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.648396 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:26.650023 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 112546us
I20260806 20:16:26.650417 1019 raft_consensus.cc:2726] T ec226f72b38a49529b88339beb36af46 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:26.651670 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.637573 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:26.655565 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 82546.5us
I20260806 20:16:26.656718 30834 leader_election.cc:302] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters:
I20260806 20:16:26.649227 30706 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:26.638974 30709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:26.574801 1374 raft_consensus.cc:686] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:26.576483 30868 rpcz_store.cc:276] 0806 20:16:22.701103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.321200 (+3620097us) service_pool.cc:227] Handling call
0806 20:16:26.342304 (+ 21104us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 20:16:26.360624 (+ 18320us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.360666 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 20:16:26.361708 (+ 1042us) log.cc:842] Serialized 78 byte log entry
0806 20:16:26.381863 (+ 20155us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:26.566879 (+185016us) log.cc:842] Serialized 14 byte log entry
0806 20:16:26.567389 (+ 510us) log.cc:842] Serialized 14 byte log entry
0806 20:16:26.568499 (+ 1110us) log.cc:842] Serialized 14 byte log entry
0806 20:16:26.568783 (+ 284us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.568896 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:26.568941 (+ 45us) raft_consensus.cc:1671] finished
0806 20:16:26.568951 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.569093 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":149687,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":8626,"raft.run_cpu_time_us":626,"raft.run_wall_time_us":7338,"spinlock_wait_cycles":3367040,"thread_start_us":202581,"threads_started":2,"wal-append.queue_time_us":87336}
W20260806 20:16:26.638276 1435 log.cc:924] Time spent T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 20:16:26.665109 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:26.624222 1348 raft_consensus.cc:482] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:26.667347 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0db3302c934290a93017bbfb29db44" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.668255 30788 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 9.
I20260806 20:16:26.668509 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
W20260806 20:16:26.670076 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 99780.6us
I20260806 20:16:26.661875 961 raft_consensus.cc:2781] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 11 FOLLOWER]: Leader election won for term 11
I20260806 20:16:26.675876 30706 raft_consensus.cc:1226] T eca12ca38e014311bbb9373e14cc903b P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:26.681905 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.682745 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0005c7c8586414a81bd94a504a598c9" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.683775 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:26.681078 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:26.687515 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1785) took 4477 ms. Trace:
I20260806 20:16:26.687951 30706 rpcz_store.cc:276] 0806 20:16:22.210215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.659284 (+4449069us) service_pool.cc:227] Handling call
0806 20:16:26.675689 (+ 16405us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.687243 (+ 11554us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.687454 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.668260 1348 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:26.685990 30834 leader_election.cc:302] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:26.693759 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.694818 31416 raft_consensus.cc:2379] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 7: Already voted for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in this term.
I20260806 20:16:26.682803 30787 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 4.
I20260806 20:16:26.697166 30787 raft_consensus.cc:1226] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 7. Current term is 8. Ops: []
W20260806 20:16:26.697168 30707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59126 (request call id 1807) took 2714 ms (client timeout 1617 ms). Trace:
W20260806 20:16:26.697993 30707 rpcz_store.cc:269] 0806 20:16:23.982391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.541153 (+558762us) service_pool.cc:227] Handling call
0806 20:16:26.696876 (+2155723us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b540035f554. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f71516732ae 00007f71565b0c27 00007f714fca4c04 00007f714fca4b92 00007f714fca4ac1 00007f714fca4a3c 00007f714fca3cf2 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:26.697060 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4205799168}
I20260806 20:16:26.698396 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1786) took 4487 ms. Trace:
I20260806 20:16:26.698805 30710 rpcz_store.cc:276] 0806 20:16:22.210488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.674059 (+4463571us) service_pool.cc:227] Handling call
0806 20:16:26.697789 (+ 23730us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.698046 (+ 257us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.698073 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.698129 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:26.698168 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.698221 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.698331 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.699826 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 408) took 4221 ms. Trace:
I20260806 20:16:26.699772 30787 raft_consensus.cc:1226] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [term 9 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260806 20:16:26.684248 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c541d91c0c4487c9a76c6fc6f81a95a" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:26.700273 30707 rpcz_store.cc:276] 0806 20:16:22.478057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.698886 (+4220829us) service_pool.cc:227] Handling call
0806 20:16:26.699272 (+ 386us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.699483 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.699508 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.699569 (+ 61us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:26.699609 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.699654 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.699753 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.700198 30788 raft_consensus.cc:2354] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9.
I20260806 20:16:26.702175 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.700057 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 404) took 5151 ms. Trace:
I20260806 20:16:26.674944 1019 raft_consensus.cc:2741] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 20:16:26.707073 30708 rpcz_store.cc:276] 0806 20:16:21.549026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.699158 (+5150132us) service_pool.cc:227] Handling call
0806 20:16:26.699529 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.699739 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.699766 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.699812 (+ 46us) raft_consensus.cc:1631] Marking committed up to 6
0806 20:16:26.699872 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.699922 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.700008 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 20:16:26.683789 1446 log.cc:924] Time spent T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260806 20:16:26.710229 30784 raft_consensus.cc:2379] T 78bef7be8f95499c84c37311213c95b6 P d4181d4d907f477f9feddc44ba3b9207 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 11: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:26.711431 30710 raft_consensus.cc:1226] T 6868aeec902a4311a782f4ef5ab064f8 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:26.633244 1337 consensus_queue.cc:260] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.678172 1286 leader_election.cc:288] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
W20260806 20:16:26.716216 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 119060us
I20260806 20:16:26.718616 30707 raft_consensus.cc:1226] T cc6e5d61a5694983a73c242e91257d86 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:26.719497 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.721025 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1793) took 3561 ms. Trace:
I20260806 20:16:26.683116 1407 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.721421 30707 rpcz_store.cc:276] 0806 20:16:23.159890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.702323 (+3542433us) service_pool.cc:227] Handling call
0806 20:16:26.718444 (+ 16121us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.720766 (+ 2322us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.720966 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.725116 30705 raft_consensus.cc:1226] T 15c9bdefd8144ce787ecb60a54ea1450 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:26.720326 30790 raft_consensus.cc:2362] T 7c541d91c0c4487c9a76c6fc6f81a95a P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 7. Re-sending same reply.
I20260806 20:16:26.728819 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1788) took 4517 ms. Trace:
I20260806 20:16:26.729112 30705 rpcz_store.cc:276] 0806 20:16:22.211023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.685886 (+4474863us) service_pool.cc:227] Handling call
0806 20:16:26.720984 (+ 35098us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.728551 (+ 7567us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.728761 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.730551 1337 raft_consensus.cc:3037] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:26.731221 30703 raft_consensus.cc:1261] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260806 20:16:26.731627 30703 pending_rounds.cc:76] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 3
I20260806 20:16:26.731877 30703 pending_rounds.cc:97] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 20:16:26.732187 30703 raft_consensus.cc:2866] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:26.733007 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1797) took 3499 ms. Trace:
I20260806 20:16:26.733392 30703 rpcz_store.cc:276] 0806 20:16:23.234175 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.708662 (+3474487us) service_pool.cc:227] Handling call
0806 20:16:26.730724 (+ 22062us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.732733 (+ 2009us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.732940 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.734061 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.734583 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1804) took 2970 ms. Trace:
I20260806 20:16:26.739125 30708 rpcz_store.cc:276] 0806 20:16:23.763923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.711020 (+2947097us) service_pool.cc:227] Handling call
0806 20:16:26.711463 (+ 443us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.718743 (+ 7280us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.718763 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.718797 (+ 34us) raft_consensus.cc:1631] Marking committed up to 3
0806 20:16:26.734364 (+ 15567us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.734429 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.734531 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.718577 1311 raft_consensus.cc:2726] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:26.747172 30711 raft_consensus.cc:1226] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:26.725183 961 raft_consensus.cc:686] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [term 11 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:26.748449 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:26.749059 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1799) took 3192 ms. Trace:
I20260806 20:16:26.749370 30711 rpcz_store.cc:276] 0806 20:16:23.556669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.713660 (+3156991us) service_pool.cc:227] Handling call
0806 20:16:26.737035 (+ 23375us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.748825 (+ 11790us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.748999 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.748409 1418 raft_consensus.cc:2781] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:26.750761 1418 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:26.751111 1418 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:26.753697 30754 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 179568us
I20260806 20:16:26.682240 1374 consensus_queue.cc:237] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.746874 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.757346 30784 raft_consensus.cc:2396] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260806 20:16:26.759868 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 89425.5us
I20260806 20:16:26.759733 30708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 414) took 1860 ms. Trace:
I20260806 20:16:26.763654 30708 rpcz_store.cc:276] 0806 20:16:24.899636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.744597 (+1844961us) service_pool.cc:227] Handling call
0806 20:16:26.756381 (+ 11784us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.756552 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.756575 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.756615 (+ 40us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:26.759477 (+ 2862us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.759550 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.759676 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.753597 986 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:26.769513 30866 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1497) took 4159 ms. Trace:
I20260806 20:16:26.770491 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.754065 961 consensus_queue.cc:237] T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.771574 30787 raft_consensus.cc:2396] T 3e24eef6446040b8abbe977d4dd5a3ca P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260806 20:16:26.772678 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.773792 30861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60894 (request call id 1495) took 4164 ms. Trace:
I20260806 20:16:26.773874 1120 consensus_queue.cc:1046] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:16:26.775031 1120 raft_consensus.cc:3032] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:26.775414 1120 raft_consensus.cc:729] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:26.776436 1120 consensus_queue.cc:260] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.777233 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:26.778110 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:26.774340 30861 rpcz_store.cc:276] 0806 20:16:22.609161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.283451 (+3674290us) service_pool.cc:227] Handling call
0806 20:16:26.293704 (+ 10253us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 20:16:26.440219 (+146515us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.440247 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 20:16:26.440871 (+ 624us) log.cc:842] Serialized 78 byte log entry
0806 20:16:26.659242 (+218371us) raft_consensus.cc:1631] Marking committed up to 2
0806 20:16:26.685365 (+ 26123us) log.cc:842] Serialized 14 byte log entry
0806 20:16:26.685597 (+ 232us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.685762 (+ 165us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:26.773522 (+ 87760us) raft_consensus.cc:1671] finished
0806 20:16:26.773544 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.773724 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58158,"prepare.queue_time_us":27778,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":12326,"raft.run_cpu_time_us":414,"raft.run_wall_time_us":13804,"thread_start_us":217911,"threads_started":1,"wal-append.queue_time_us":287097}
I20260806 20:16:26.778610 30785 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33236 (request call id 1520) took 3082 ms. Trace:
I20260806 20:16:26.779138 30756 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:26.779297 1120 raft_consensus.cc:3037] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:26.780148 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:26.780166 1019 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P d4181d4d907f477f9feddc44ba3b9207 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 20:16:26.769951 30866 rpcz_store.cc:276] 0806 20:16:22.609675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.299295 (+3689620us) service_pool.cc:227] Handling call
0806 20:16:26.299955 (+ 660us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:26.341118 (+ 41163us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.341157 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:26.366364 (+ 25207us) log.cc:842] Serialized 26 byte log entry
0806 20:16:26.460539 (+ 94175us) raft_consensus.cc:1631] Marking committed up to 5
0806 20:16:26.662826 (+202287us) log.cc:842] Serialized 14 byte log entry
0806 20:16:26.663085 (+ 259us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.663153 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:26.769292 (+106139us) raft_consensus.cc:1671] finished
0806 20:16:26.769310 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.769439 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13105,"prepare.queue_time_us":62661,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":41606,"raft.run_cpu_time_us":544,"raft.run_wall_time_us":24968,"thread_start_us":270996,"threads_started":2,"wal-append.queue_time_us":196581}
I20260806 20:16:26.774135 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:26.785914 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.786300 31420 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:26.778985 30834 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:26.782743 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:26.786671 30790 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 9.
I20260806 20:16:26.779026 30785 rpcz_store.cc:276] 0806 20:16:23.696125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:24.710446 (+1014321us) service_pool.cc:227] Handling call
0806 20:16:26.411477 (+1701031us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.456358 (+ 44881us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.778375 (+322017us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54000573d0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:26.778531 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3816988544}
I20260806 20:16:26.779604 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:26.794203 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:26.795020 30866 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 9.
I20260806 20:16:26.796262 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.797724 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.796221 30786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39c6ca59dfb42bda66c72b0bcf52faa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:26.798339 30792 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 4.
I20260806 20:16:26.798625 30708 raft_consensus.cc:1226] T 412dab7986704f48be75b1fdbdccc9fa P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 20:16:26.798522 30786 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:26.799258 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.800282 30792 raft_consensus.cc:2379] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 8: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:26.801111 30792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca12ca38e014311bbb9373e14cc903b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:26.801756 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:26.801757 30792 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:26.803215 30711 raft_consensus.cc:1226] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:26.806337 30711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 416) took 1811 ms. Trace:
I20260806 20:16:26.806654 30711 rpcz_store.cc:276] 0806 20:16:24.995165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.758881 (+1763716us) service_pool.cc:227] Handling call
0806 20:16:26.802997 (+ 44116us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.806092 (+ 3095us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.806283 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.807973 30711 raft_consensus.cc:1261] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:16:26.810464 30711 raft_consensus.cc:1261] T cd962fc44d164b8b80175c097e321044 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260806 20:16:26.803227 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.812983 30785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.805567 30787 raft_consensus.cc:2454] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 9.
I20260806 20:16:26.814256 1301 leader_election.cc:288] T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:26.814455 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd962fc44d164b8b80175c097e321044" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:26.816248 30835 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:26.817250 1311 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:26.816931 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:26.818923 30787 raft_consensus.cc:2379] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 7: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:26.820096 30787 raft_consensus.cc:1226] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:26.787074 1361 raft_consensus.cc:482] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:26.826810 1361 raft_consensus.cc:504] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.802578 30708 raft_consensus.cc:1226] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260806 20:16:26.831374 30707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1806) took 2849 ms. Trace:
I20260806 20:16:26.831763 30707 rpcz_store.cc:276] 0806 20:16:23.981973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.735958 (+2753985us) service_pool.cc:227] Handling call
0806 20:16:26.760499 (+ 24541us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.777933 (+ 17434us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.786227 (+ 8294us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:26.831120 (+ 44893us) raft_consensus.cc:1631] Marking committed up to 4
0806 20:16:26.831158 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.831214 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.831321 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.834270 30712 raft_consensus.cc:2454] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in term 7.
I20260806 20:16:26.835122 30706 raft_consensus.cc:1226] T 0f3393f94b2649378cb19b57fecfb21e P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 7. Ops: [5.7-5.7]
I20260806 20:16:26.830156 30786 raft_consensus.cc:2454] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 10.
I20260806 20:16:26.837081 30792 raft_consensus.cc:2454] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 7.
I20260806 20:16:26.826712 30785 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 8.
I20260806 20:16:26.840313 30705 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P 42cabca48db7403b8523a3382a046043 [term 10 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 9. Current term is 10. Ops: []
I20260806 20:16:26.844883 30705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 409) took 2210 ms. Trace:
I20260806 20:16:26.845371 30705 rpcz_store.cc:276] 0806 20:16:24.634117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.749178 (+2115061us) service_pool.cc:227] Handling call
0806 20:16:26.840118 (+ 90940us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.844575 (+ 4457us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.844813 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 20:16:26.848438 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:26.861554 1411 raft_consensus.cc:3032] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 20:16:26.862080 1411 raft_consensus.cc:729] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:26.863099 1411 consensus_queue.cc:260] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.863868 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8d5f5ccc46476aac7a641abae6bdfa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:26.864368 1411 raft_consensus.cc:3037] T 28d6e84eb2874b2b9dca3387b3d98eff P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:26.866057 987 raft_consensus.cc:504] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.867719 30868 raft_consensus.cc:2354] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 4. Current term is 5.
I20260806 20:16:26.866905 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:26.869532 30866 raft_consensus.cc:2379] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 9: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:26.868278 987 leader_election.cc:288] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:26.873127 30756 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:26.843613 1311 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:26.874272 1286 raft_consensus.cc:2781] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:26.874780 1286 raft_consensus.cc:482] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:26.875185 1286 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:26.876192 1041 consensus_queue.cc:1046] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 20:16:26.877231 1041 raft_consensus.cc:3032] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 LEADER]: Stepping down as leader of term 10
I20260806 20:16:26.877614 1041 raft_consensus.cc:729] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
W20260806 20:16:26.878881 1444 log.cc:924] Time spent T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.247s user 0.002s sys 0.000s
I20260806 20:16:26.878639 1041 consensus_queue.cc:260] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.880167 1041 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 FOLLOWER]: Advancing to term 11
I20260806 20:16:26.881289 30756 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:26.881615 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 89795.8us
I20260806 20:16:26.882269 30710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1792) took 3755 ms. Trace:
I20260806 20:16:26.882891 1407 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:26.899977 30710 rpcz_store.cc:276] 0806 20:16:23.126495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.699734 (+3573239us) service_pool.cc:227] Handling call
0806 20:16:26.702637 (+ 2903us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.713756 (+ 11119us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.882042 (+168286us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b54000c6f50. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:26.882185 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5836672}
I20260806 20:16:26.885152 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:26.886233 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:26.887553 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f92e9dc9db864f0b8e238a39be31b12e" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:26.870565 1445 log.cc:924] Time spent T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.248s user 0.002s sys 0.000s
I20260806 20:16:26.889189 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:26.905364 30866 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 8.
I20260806 20:16:26.917747 30706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59126 (request call id 1791) took 4639 ms. Trace:
I20260806 20:16:26.918208 30706 rpcz_store.cc:276] 0806 20:16:22.278501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.688677 (+4410176us) service_pool.cc:227] Handling call
0806 20:16:26.834863 (+146186us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:26.839794 (+ 4931us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.917518 (+ 77724us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400030cd0. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4471 00007f715166ea09 00007f71565b02e9 00007f714fca2684 00007f714fca2612 00007f714fca2541 00007f714fca24bc 00007f714fca1772 00007f714faa594c 00007f714faa4d86 00007f714faa472f
0806 20:16:26.917672 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":262752512}
W20260806 20:16:26.928167 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 47019.4us
I20260806 20:16:26.928946 30835 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:26.942426 30756 leader_election.cc:302] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:26.942754 30784 raft_consensus.cc:1226] T e39c6ca59dfb42bda66c72b0bcf52faa P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 9. Current term is 10. Ops: []
W20260806 20:16:26.943742 30757 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.294s (SENT) [suppressed 50 similar messages]
I20260806 20:16:26.955389 1356 raft_consensus.cc:2726] T eca12ca38e014311bbb9373e14cc903b P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:16:26.962075 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 65672.2us
I20260806 20:16:26.962703 1407 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 20:16:27.017706 1407 raft_consensus.cc:482] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 20:16:27.017833 30673 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 80235us
I20260806 20:16:27.018605 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.017118 30835 leader_election.cc:302] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.020413 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.018146 1407 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:27.021528 1337 raft_consensus.cc:2726] T c0005c7c8586414a81bd94a504a598c9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:27.022315 1337 consensus_queue.cc:1046] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:16:27.023533 1337 raft_consensus.cc:3032] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:27.023901 1337 raft_consensus.cc:729] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.024469 30835 leader_election.cc:398] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9.
I20260806 20:16:27.024928 30835 leader_election.cc:401] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260806 20:16:27.025663 1309 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:26.940995 1286 raft_consensus.cc:504] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
W20260806 20:16:27.026333 1305 consensus_meta.cc:287] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207: Time spent flushing consensus metadata: real 0.546s user 0.017s sys 0.000s
I20260806 20:16:27.026773 1305 raft_consensus.cc:504] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:26.987237 30869 raft_consensus.cc:2396] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260806 20:16:27.028139 30789 raft_consensus.cc:2379] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 in current term 8: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:27.028724 30792 raft_consensus.cc:1226] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 5. Current term is 6. Ops: []
I20260806 20:16:27.029031 1305 leader_election.cc:288] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:27.030109 1398 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:26.964720 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:27.031654 30788 raft_consensus.cc:2396] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58f2f6256ee14cb38a1dbcbf5bb590a0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 20:16:27.032905 31416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:27.033308 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:27.032264 1398 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:27.033931 31414 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:26.971392 30832 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.196:35519, user_credentials={real_user=slave}} blocked reactor thread for 57798.5us
I20260806 20:16:26.987934 1336 consensus_queue.cc:1046] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260806 20:16:26.988605 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 46317.2us
W20260806 20:16:26.987787 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 208925us
I20260806 20:16:26.933511 1019 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:26.992205 30709 raft_consensus.cc:3037] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:26.999485 30867 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45134 (request call id 262) took 3071 ms. Trace:
I20260806 20:16:26.998569 30603 catalog_manager.cc:5495] T 5ba1969a3911423e8e51fc4f2962515d P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 6 to 7, leader changed from <none> to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 7 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:27.018661 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0911118ae8e247aca188955eb02c8728" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.019639 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.018124 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:26.991475 30713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 403) took 5520 ms. Trace:
I20260806 20:16:26.989339 1361 leader_election.cc:288] T 33e2a287c4de469eb5365fa7cdc6b409 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:26.987551 31388 leader_election.cc:302] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters:
I20260806 20:16:27.037657 30713 rpcz_store.cc:276] 0806 20:16:21.471505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.523592 (+5052087us) service_pool.cc:227] Handling call
0806 20:16:26.575496 (+ 51904us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:26.627501 (+ 52005us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.627550 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:26.628011 (+ 461us) log.cc:842] Serialized 26 byte log entry
0806 20:16:26.746539 (+118528us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:26.746577 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.746649 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:26.991234 (+244585us) raft_consensus.cc:1671] finished
0806 20:16:26.991264 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.991411 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7047,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":137,"thread_start_us":118082,"threads_started":1,"wal-append.queue_time_us":299721}
I20260806 20:16:27.037267 1019 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:27.061887 30713 raft_consensus.cc:1226] T 95c5938e508b4075b878ae1f5ea726d6 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Rejecting Update request from peer d4181d4d907f477f9feddc44ba3b9207 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:27.064553 30709 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9.
I20260806 20:16:27.055639 30835 leader_election.cc:302] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.056540 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:27.065659 30788 raft_consensus.cc:3037] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:27.036962 30867 rpcz_store.cc:276] 0806 20:16:23.928518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.438425 (+2509907us) service_pool.cc:227] Handling call
0806 20:16:26.438902 (+ 477us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 20:16:26.439433 (+ 531us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 20:16:26.439467 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 20:16:26.440007 (+ 540us) log.cc:842] Serialized 26 byte log entry
0806 20:16:26.568126 (+128119us) raft_consensus.cc:1631] Marking committed up to 8
0806 20:16:26.568160 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:26.568241 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 20:16:26.869644 (+301403us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b44000232cc. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f714e597f11 00007f714e665a95 00007f714e6664b2 00007f7151302511 00007f71512fde1a 00007f71512f1bee 00007f71512f0f36 00007f71512f0b58 00007f71512fa551 00007f71512fa509 00007f71512fa499
0806 20:16:26.999110 (+129466us) raft_consensus.cc:1671] finished
0806 20:16:26.999133 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:26.999392 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":441,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":151,"spinlock_wait_cycles":100391168,"thread_start_us":127736,"threads_started":1,"wal-append.queue_time_us":181511}
I20260806 20:16:27.035027 1286 leader_election.cc:288] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:27.070734 30834 leader_election.cc:302] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.072513 31420 raft_consensus.cc:2454] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:27.074633 30834 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.077435 30754 leader_election.cc:302] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.077718 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:27.067606 1019 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:27.077087 987 raft_consensus.cc:2726] T 47b5901783b8423c9e494c131b0a64e9 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:27.068817 1311 raft_consensus.cc:3037] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Advancing to term 11
I20260806 20:16:27.024783 1337 consensus_queue.cc:260] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.086499 30834 leader_election.cc:302] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:27.085951 30676 connection.cc:232] RPC call timeout handler was delayed by -2.1178338160098065s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
W20260806 20:16:27.099197 30676 connection.cc:232] RPC call timeout handler was delayed by -2.0899338159943s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1579s for a response
W20260806 20:16:27.099522 30676 connection.cc:232] RPC call timeout handler was delayed by -2.0692338160006329s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
W20260806 20:16:27.099813 30676 connection.cc:232] RPC call timeout handler was delayed by -2.0188338159932755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
W20260806 20:16:27.100075 30676 connection.cc:232] RPC call timeout handler was delayed by -1.9180338160076644s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
W20260806 20:16:27.100332 30676 connection.cc:232] RPC call timeout handler was delayed by -1.9162338160094805s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
W20260806 20:16:27.100584 30676 connection.cc:232] RPC call timeout handler was delayed by -1.8217338160029612s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1877s for a response
W20260806 20:16:27.100849 30676 connection.cc:232] RPC call timeout handler was delayed by -1.8154338159947656s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
W20260806 20:16:27.101105 30676 connection.cc:232] RPC call timeout handler was delayed by -1.7740338160074316s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20260806 20:16:27.101362 30676 connection.cc:232] RPC call timeout handler was delayed by -1.747033816005569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.196s for a response
W20260806 20:16:27.101966 30676 connection.cc:232] RPC call timeout handler was delayed by -1.7398338160128333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
W20260806 20:16:27.102250 30676 connection.cc:232] RPC call timeout handler was delayed by -1.6210338159871753s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
I20260806 20:16:27.082348 1286 raft_consensus.cc:2781] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:27.104349 30756 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.084427 878 raft_consensus.cc:2781] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 20:16:27.086700 1398 raft_consensus.cc:482] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:27.106089 30603 catalog_manager.cc:5495] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 6 to 8, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 8 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:27.121153 878 raft_consensus.cc:686] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.111500 1361 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:27.126919 30676 leader_election.cc:302] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.091609 1348 raft_consensus.cc:504] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.120263 987 raft_consensus.cc:2781] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:16:27.120057 1286 raft_consensus.cc:686] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:27.136489 30756 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:27.133639 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 114255us
W20260806 20:16:27.135568 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 148089us
I20260806 20:16:27.139261 30711 raft_consensus.cc:1226] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 20:16:27.127811 1361 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:27.118878 1398 raft_consensus.cc:504] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.143735 30834 leader_election.cc:302] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:27.134246 30603 catalog_manager.cc:5495] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 4 to 6, leader changed from d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194) to 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196). New cstate: current_term: 6 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:27.130084 1411 consensus_queue.cc:1046] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260806 20:16:27.138293 987 raft_consensus.cc:686] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.148363 1361 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:27.145467 1374 raft_consensus.cc:2781] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 20:16:27.152355 1337 raft_consensus.cc:3037] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:27.162992 986 raft_consensus.cc:2726] T 3070a7930e6d47cd8776055aab6539a3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 20:16:27.179306 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 99305.4us
I20260806 20:16:27.180382 30835 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.158684 30603 catalog_manager.cc:5495] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:27.182865 30788 raft_consensus.cc:2454] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 5.
W20260806 20:16:27.182608 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 64730.6us
I20260806 20:16:27.170270 1374 raft_consensus.cc:482] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:27.169688 1293 raft_consensus.cc:2726] T 90f7b0d94b4e4047a32e7fbdde6c35d5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 20:16:27.198940 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 95c5938e508b4075b878ae1f5ea726d6, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:27.203763 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 95c5938e508b4075b878ae1f5ea726d6, TS 58d59beb597f4ed99d84d102ddb41ff5 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.155193 1407 raft_consensus.cc:504] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.201565 30703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41700 (request call id 413) took 2382 ms. Trace:
I20260806 20:16:27.196261 30676 leader_election.cc:302] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.200405 31414 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
W20260806 20:16:27.207363 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 52311.3us
I20260806 20:16:27.192787 30603 catalog_manager.cc:5495] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 20:16:27.145766 878 consensus_queue.cc:237] T 3070a7930e6d47cd8776055aab6539a3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.210865 961 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:27.213243 30754 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.155838 1418 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.213231 31389 leader_election.cc:302] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.213598 30603 catalog_manager.cc:5495] T 01fd0995e74a449090ea2b00062f625e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
I20260806 20:16:27.204885 1374 raft_consensus.cc:3037] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:27.227499 30756 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:27.206696 30703 rpcz_store.cc:276] 0806 20:16:24.819300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:26.744630 (+1925330us) service_pool.cc:227] Handling call
0806 20:16:26.830195 (+ 85565us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 20:16:26.873091 (+ 42896us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 20:16:27.097726 (+224635us) log.cc:842] Serialized 14 byte log entry
0806 20:16:27.192824 (+ 95098us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 20:16:27.192877 (+ 53us) raft_consensus.cc:1631] Marking committed up to 7
0806 20:16:27.201020 (+ 8143us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 20:16:27.201187 (+ 167us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 20:16:27.201413 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":263180,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":22076,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":10028,"thread_start_us":266499,"threads_started":2}
I20260806 20:16:27.212355 1398 leader_election.cc:288] T bf7304ca0290443b884297d0e8744607 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:27.228466 1348 leader_election.cc:288] T 24eb40e43f13412e8f080b6b6ca292de P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 election: Requested vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:27.239253 31422 raft_consensus.cc:1226] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 5. Current term is 6. Ops: []
W20260806 20:16:27.240206 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 61947.7us
I20260806 20:16:27.243718 30603 catalog_manager.cc:5495] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 reported cstate change: term changed from 3 to 5, leader changed from 42cabca48db7403b8523a3382a046043 (127.28.186.193) to 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195). New cstate: current_term: 5 leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } } }
W20260806 20:16:27.242264 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 66197.8us
W20260806 20:16:27.236992 1453 log.cc:924] Time spent T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
W20260806 20:16:27.248726 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 55505.6us
I20260806 20:16:27.240432 1239 raft_consensus.cc:2781] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 20:16:27.242201 986 raft_consensus.cc:2726] T 8ec60d6f5b6c486b80781ad3cb85a06b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:27.258038 1239 raft_consensus.cc:482] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58f2f6256ee14cb38a1dbcbf5bb590a0)
I20260806 20:16:27.260787 1239 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Advancing to term 7
W20260806 20:16:27.248155 1297 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.003s sys 0.013s
I20260806 20:16:27.229522 1336 consensus_queue.cc:1046] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260806 20:16:27.208557 1286 consensus_queue.cc:237] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.268815 31389 leader_election.cc:302] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0; no voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207
W20260806 20:16:27.269157 30601 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:41560 (request call id 218) took 20798 ms (client timeout 15000 ms). Trace:
I20260806 20:16:27.270991 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0607668d3924d5cba6acd89b2cdc25f" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
W20260806 20:16:27.271642 1459 log.cc:924] Time spent T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 20:16:27.272514 1309 raft_consensus.cc:2726] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029) with higher term. Message: Invalid argument: T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 8. Current term is 9.
I20260806 20:16:27.273538 1400 raft_consensus.cc:2781] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 20:16:27.274072 1400 raft_consensus.cc:686] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:16:27.274566 1455 log.cc:924] Time spent T b060012888ff4d76b254b43d1f1959bb P d4181d4d907f477f9feddc44ba3b9207: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20260806 20:16:27.274793 1019 raft_consensus.cc:504] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.275012 1400 consensus_queue.cc:237] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.267644 1463 raft_consensus.cc:2781] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 20:16:27.263873 1297 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.681s user 0.004s sys 0.014s
I20260806 20:16:27.273957 30703 raft_consensus.cc:1226] T f53758bb873349479f86e0abf34c738b P 42cabca48db7403b8523a3382a046043 [term 11 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 10. Current term is 11. Ops: []
W20260806 20:16:27.273317 1458 log.cc:924] Time spent T f53758bb873349479f86e0abf34c738b P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260806 20:16:27.295169 31420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eb40e43f13412e8f080b6b6ca292de" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:27.264835 1384 raft_consensus.cc:2726] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:27.239369 987 consensus_queue.cc:237] T 67c52f63d3e8412bbd0e51fe5b81568b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.283820 1418 leader_election.cc:288] T a5e69bf76cc845c8822b4cfab14eac47 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [CANDIDATE]: Term 6 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
W20260806 20:16:27.325871 30756 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 117073us
I20260806 20:16:27.281600 1019 leader_election.cc:288] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:27.334285 31414 raft_consensus.cc:1226] T 47c09420eb1e4f3692c165f76cd1a7d4 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:27.339653 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 70890.4us
W20260806 20:16:27.340253 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 43981.6us
I20260806 20:16:27.347359 1411 raft_consensus.cc:3032] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 20:16:27.347782 1411 raft_consensus.cc:729] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:27.347128 1337 consensus_queue.cc:1046] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 20:16:27.348702 1411 consensus_queue.cc:260] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.352850 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e69bf76cc845c8822b4cfab14eac47" candidate_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:27.365281 1461 raft_consensus.cc:3032] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:27.365820 1461 raft_consensus.cc:729] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:27.370599 30676 leader_election.cc:302] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
W20260806 20:16:27.385185 1041 consensus_meta.cc:287] T 78bef7be8f95499c84c37311213c95b6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Time spent flushing consensus metadata: real 0.505s user 0.011s sys 0.000s
W20260806 20:16:27.388348 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 208053us
I20260806 20:16:27.346582 1384 raft_consensus.cc:2726] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 20:16:27.293952 1474 consensus_queue.cc:1046] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260806 20:16:27.363251 1407 leader_election.cc:288] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
W20260806 20:16:27.398877 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 46305.5us
I20260806 20:16:27.358023 1467 consensus_queue.cc:1046] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260806 20:16:27.406881 30835 leader_election.cc:302] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:27.405835 30676 leader_election.cc:302] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.402290 961 raft_consensus.cc:2726] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 20:16:27.396660 986 raft_consensus.cc:2781] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:27.410506 30756 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.414390 1361 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.431752 30756 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.415189 1374 raft_consensus.cc:504] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.433627 1305 raft_consensus.cc:2741] T 26a3d1fc21e040c09103cfcf9e539659 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 20:16:27.418412 31414 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260806 20:16:27.269989 30601 rpcz_store.cc:269] 0806 20:16:06.439641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 20:16:06.439941 (+ 300us) service_pool.cc:227] Handling call
0806 20:16:06.714240 (+274299us) spinlock_profiling.cc:243] Waited 69 ms on lock 0x7b5400002dd4. stack: 00007f714e721033 00007f714e720f30 00007f714e07f534 00007f7157d7f8b4 00007f7157d7f849 00007f71515f4af0 00007f7151677f2a 00007f71575c9d6c 00007f71576c3c3e 00007f71559d5fb4 00007f71559d5f42 00007f71559d5e71 00007f71559d5dec 00007f71559d50a2 00007f714faa594c 00007f714faa4d86
0806 20:16:27.234630 (+20520390us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 20:16:22.519000 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 20:16:22.977661 (+458661us) write_op.cc:432] Acquiring schema lock in shared mode
0806 20:16:22.977715 (+ 54us) write_op.cc:435] Acquired schema lock
0806 20:16:22.977725 (+ 10us) tablet.cc:660] Decoding operations
0806 20:16:22.978690 (+ 965us) write_op.cc:620] Acquiring the partition lock for write op
0806 20:16:22.978789 (+ 99us) write_op.cc:641] Partition lock acquired for write op
0806 20:16:22.978816 (+ 27us) tablet.cc:683] Acquiring locks for 7 operations
0806 20:16:22.979516 (+ 700us) tablet.cc:699] Row locks acquired
0806 20:16:22.979550 (+ 34us) write_op.cc:260] PREPARE: finished
0806 20:16:22.979742 (+ 192us) write_op.cc:270] Start()
0806 20:16:22.979906 (+ 164us) write_op.cc:276] Timestamp: P: 1786047382979667 usec, L: 0
0806 20:16:22.979922 (+ 16us) op_driver.cc:331] REPLICATION: starting
0806 20:16:22.980720 (+ 798us) log.cc:842] Serialized 2882 byte log entry
0806 20:16:25.001583 (+2020863us) op_driver.cc:444] REPLICATION: finished
0806 20:16:25.875498 (+873915us) write_op.cc:301] APPLY: starting
0806 20:16:25.875586 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
0806 20:16:25.875789 (+ 203us) tablet.cc:1368] finished BulkCheckPresence
0806 20:16:25.875924 (+ 135us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 20:16:25.876659 (+ 735us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 20:16:25.905912 (+ 29253us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0806 20:16:25.914135 (+ 8223us) write_op.cc:312] APPLY: finished
0806 20:16:25.964874 (+ 50739us) log.cc:842] Serialized 59 byte log entry
0806 20:16:26.176449 (+211575us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:26.178560 (+ 2111us) write_op.cc:454] Released schema lock
0806 20:16:26.180465 (+ 1905us) write_op.cc:341] FINISH: Updating metrics
0806 20:16:26.186366 (+ 5901us) write_op.cc:489] Releasing partition, row and schema locks
0806 20:16:26.186397 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":823347,"reported_tablets":49,"spinlock_wait_cycles":158781312,"child_traces":[["op",{"apply.queue_time_us":873636,"apply.run_cpu_time_us":20626,"apply.run_wall_time_us":311223,"num_ops":7,"prepare.queue_time_us":474231,"prepare.run_cpu_time_us":74217,"prepare.run_wall_time_us":1374922,"raft.queue_time_us":412686,"raft.run_cpu_time_us":33790,"raft.run_wall_time_us":1434964,"replication_time_us":2021564,"thread_start_us":2854006,"threads_started":5,"wal-append.queue_time_us":1184821,"wal-append.run_cpu_time_us":26383,"wal-append.run_wall_time_us":2927038}]]}
I20260806 20:16:27.434845 1305 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:27.435242 1305 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:27.435629 1305 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:27.435732 1390 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:16:27.435073 986 raft_consensus.cc:482] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:27.438051 986 raft_consensus.cc:3037] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:27.387368 1293 consensus_queue.cc:1046] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260806 20:16:27.419796 1461 consensus_queue.cc:260] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.433130 30756 leader_election.cc:302] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:27.409283 1411 raft_consensus.cc:3037] T e39c6ca59dfb42bda66c72b0bcf52faa P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:27.397185 1463 raft_consensus.cc:686] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 LEADER]: Becoming Leader. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:27.431813 1361 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:27.438205 1473 raft_consensus.cc:3032] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 20:16:27.458798 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 53073.2us
W20260806 20:16:27.460021 31388 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 49639.5us
I20260806 20:16:27.447548 961 raft_consensus.cc:2781] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 20:16:27.468598 1468 log.cc:924] Time spent T f92e9dc9db864f0b8e238a39be31b12e P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20260806 20:16:27.469295 1374 leader_election.cc:288] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:27.454329 1356 raft_consensus.cc:2726] T 412dab7986704f48be75b1fdbdccc9fa P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 20:16:27.460915 1473 raft_consensus.cc:729] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.470700 30707 raft_consensus.cc:1226] T 67c52f63d3e8412bbd0e51fe5b81568b P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 8. Current term is 9. Ops: []
I20260806 20:16:27.409435 1019 consensus_queue.cc:1046] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20260806 20:16:27.471724 1019 raft_consensus.cc:3032] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 20:16:27.472127 1019 raft_consensus.cc:729] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
W20260806 20:16:27.472357 1483 log.cc:924] Time spent T e0607668d3924d5cba6acd89b2cdc25f P 42cabca48db7403b8523a3382a046043: Append to log took a long time: real 0.059s user 0.002s sys 0.000s
I20260806 20:16:27.409277 878 consensus_queue.cc:1046] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260806 20:16:27.471261 961 raft_consensus.cc:686] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.494525 30786 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0911118ae8e247aca188955eb02c8728"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:27.488852 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:27.498416 961 consensus_queue.cc:237] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.472259 1479 consensus_queue.cc:1046] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
W20260806 20:16:27.502417 1477 log.cc:924] Time spent T b060012888ff4d76b254b43d1f1959bb P 58d59beb597f4ed99d84d102ddb41ff5: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20260806 20:16:27.470804 1482 log.cc:924] Time spent T 88608755c61d42ec8ccc0c3c8e536bd6 P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.051s user 0.000s sys 0.003s
I20260806 20:16:27.465600 1398 consensus_queue.cc:1046] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:16:27.504086 1398 raft_consensus.cc:3032] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:27.504469 1398 raft_consensus.cc:729] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
W20260806 20:16:27.473260 30834 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.193:42937, user_credentials={real_user=slave}} blocked reactor thread for 47518.1us
I20260806 20:16:27.508949 1407 consensus_queue.cc:1035] T bf7304ca0290443b884297d0e8744607 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:27.490588 1309 consensus_queue.cc:1046] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260806 20:16:27.510430 1293 consensus_queue.cc:1046] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:16:27.511266 1309 raft_consensus.cc:3032] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:27.511636 1309 raft_consensus.cc:729] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.512171 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.446354 1348 raft_consensus.cc:3037] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:16:27.495087 1461 raft_consensus.cc:3037] T 67c52f63d3e8412bbd0e51fe5b81568b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:27.518465 878 raft_consensus.cc:3032] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:27.518915 878 raft_consensus.cc:729] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.520015 878 consensus_queue.cc:260] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.474831 30676 leader_election.cc:302] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42cabca48db7403b8523a3382a046043; no voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.522019 1356 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:27.522250 1417 raft_consensus.cc:2726] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 20:16:27.522682 1356 raft_consensus.cc:504] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:27.523386 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 48665.2us
I20260806 20:16:27.505295 1398 consensus_queue.cc:260] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.524262 30676 leader_election.cc:302] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:27.524850 1356 leader_election.cc:288] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:27.525089 1417 raft_consensus.cc:2781] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 20:16:27.511574 1293 raft_consensus.cc:3032] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 20:16:27.525168 1398 raft_consensus.cc:3037] T 47c09420eb1e4f3692c165f76cd1a7d4 P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:27.525718 1417 raft_consensus.cc:482] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:27.522367 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.526257 1417 raft_consensus.cc:3037] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:27.526491 30863 raft_consensus.cc:2362] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9. Re-sending same reply.
I20260806 20:16:27.526710 1293 raft_consensus.cc:729] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.527596 1293 consensus_queue.cc:260] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.528493 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc53de31500c4908b8ec8ee3a0f7498c" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.529157 30867 raft_consensus.cc:2454] T dc53de31500c4908b8ec8ee3a0f7498c P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:27.530583 30861 raft_consensus.cc:1261] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260806 20:16:27.530943 30861 pending_rounds.cc:76] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting all ops after (but not including) 3
I20260806 20:16:27.531140 30861 pending_rounds.cc:97] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260806 20:16:27.531406 30861 raft_consensus.cc:2866] T cd962fc44d164b8b80175c097e321044 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:27.534328 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.535771 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.537864 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6e5d61a5694983a73c242e91257d86" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.539080 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6251f4f1aa48ca869b81256b9fd89a" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:27.539280 1478 log.cc:924] Time spent T 412dab7986704f48be75b1fdbdccc9fa P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20260806 20:16:27.540129 30863 raft_consensus.cc:2454] T 7e6251f4f1aa48ca869b81256b9fd89a P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 5.
I20260806 20:16:27.540856 1480 raft_consensus.cc:3032] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:27.541324 1480 raft_consensus.cc:729] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.542148 1480 consensus_queue.cc:260] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.543567 1480 raft_consensus.cc:3037] T a5e69bf76cc845c8822b4cfab14eac47 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
W20260806 20:16:27.545773 30757 leader_election.cc:334] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.719s (SENT)
I20260806 20:16:27.546309 30757 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:27.540148 1475 raft_consensus.cc:3032] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:27.547219 1475 raft_consensus.cc:729] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:27.548022 1475 consensus_queue.cc:260] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.549283 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b273ef36e4c6eb6873cc61471cb53" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.549585 1475 raft_consensus.cc:3037] T 24eb40e43f13412e8f080b6b6ca292de P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:27.549899 1297 raft_consensus.cc:3032] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 20:16:27.550197 30861 raft_consensus.cc:2379] T 672b273ef36e4c6eb6873cc61471cb53 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 7: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:27.521320 878 raft_consensus.cc:3037] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:27.551457 1417 raft_consensus.cc:504] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.555307 30676 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:27.555956 1311 raft_consensus.cc:2726] T 78bef7be8f95499c84c37311213c95b6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 20:16:27.558812 1348 raft_consensus.cc:2726] T 28d6e84eb2874b2b9dca3387b3d98eff P 42cabca48db7403b8523a3382a046043 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 20:16:27.529184 1293 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:27.559589 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.550242 1297 raft_consensus.cc:729] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:27.561471 1479 raft_consensus.cc:3032] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 LEADER]: Stepping down as leader of term 10
I20260806 20:16:27.566121 30866 raft_consensus.cc:2454] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:27.576692 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.573908 1348 raft_consensus.cc:2781] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election won for term 8
W20260806 20:16:27.583837 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 179849us
I20260806 20:16:27.569825 1479 raft_consensus.cc:729] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:27.586074 30756 leader_election.cc:302] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.587927 30835 leader_election.cc:302] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5; no voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207
I20260806 20:16:27.588016 1286 raft_consensus.cc:2781] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 20:16:27.588641 1286 raft_consensus.cc:686] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [term 9 LEADER]: Becoming Leader. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:27.589620 1286 consensus_queue.cc:237] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.593117 30710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:27.594892 30707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb9334179bc40b1b3868fc4773af362" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:27.596267 30711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:27.596925 30711 raft_consensus.cc:3037] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 6 FOLLOWER]: Advancing to term 8
I20260806 20:16:27.602406 30703 raft_consensus.cc:1261] T 8a0db3302c934290a93017bbfb29db44 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 20:16:27.604537 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:27.605886 30708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1485b55849242feae0e8b2b3b28307f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:27.607278 30713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:27.608953 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:27.610109 30709 raft_consensus.cc:1261] T 942abb68076e47a0a24672bb92649892 P 42cabca48db7403b8523a3382a046043 [term 7 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 20:16:27.611277 30712 raft_consensus.cc:2454] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9.
I20260806 20:16:27.612854 30706 raft_consensus.cc:1261] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260806 20:16:27.613253 30706 pending_rounds.cc:76] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: Aborting all ops after (but not including) 4
I20260806 20:16:27.613472 30706 pending_rounds.cc:97] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260806 20:16:27.613704 30706 raft_consensus.cc:2866] T 9f32336173df4611829be37d396e86ba P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 20:16:27.625665 30711 raft_consensus.cc:2454] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:27.626685 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1969a3911423e8e51fc4f2962515d" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:27.626569 31389 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 122234us
I20260806 20:16:27.627426 30861 raft_consensus.cc:2454] T 5ba1969a3911423e8e51fc4f2962515d P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:27.582357 1400 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:27.579109 1417 leader_election.cc:288] T cca4a581dd9f4cff973793835675b6c3 P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 8 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:27.573657 1374 consensus_queue.cc:1035] T 8ec60d6f5b6c486b80781ad3cb85a06b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260806 20:16:27.512526 1309 consensus_queue.cc:260] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.585094 1356 raft_consensus.cc:2726] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260806 20:16:27.634555 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 46715.9us
I20260806 20:16:27.635430 30835 leader_election.cc:302] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.579877 1390 consensus_queue.cc:1046] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.048s
I20260806 20:16:27.649600 1463 consensus_queue.cc:237] T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.626569 1348 raft_consensus.cc:686] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [term 8 LEADER]: Becoming Leader. State: Replica: 42cabca48db7403b8523a3382a046043, State: Running, Role: LEADER
I20260806 20:16:27.645686 1019 consensus_queue.cc:260] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.654947 1348 consensus_queue.cc:237] T dc53de31500c4908b8ec8ee3a0f7498c P 42cabca48db7403b8523a3382a046043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.655980 1019 raft_consensus.cc:3037] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:27.658090 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761e8328b9d247fcb1d00f0aeff373f0" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.661831 30863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e24eef6446040b8abbe977d4dd5a3ca" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.662563 30863 raft_consensus.cc:2454] T 3e24eef6446040b8abbe977d4dd5a3ca P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:27.583561 1361 consensus_queue.cc:1035] T cd962fc44d164b8b80175c097e321044 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260806 20:16:27.668642 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3070a7930e6d47cd8776055aab6539a3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.668710 1297 consensus_queue.cc:260] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.671970 1297 raft_consensus.cc:3037] T 7cb9334179bc40b1b3868fc4773af362 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:27.471273 1473 consensus_queue.cc:260] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.673168 1473 raft_consensus.cc:3037] T eca12ca38e014311bbb9373e14cc903b P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:27.675372 1390 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:27.675870 1390 raft_consensus.cc:504] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.661897 1309 raft_consensus.cc:3037] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Advancing to term 6
I20260806 20:16:27.679941 1479 consensus_queue.cc:260] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.681797 1361 raft_consensus.cc:3032] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:27.682219 1361 raft_consensus.cc:729] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d4181d4d907f477f9feddc44ba3b9207, State: Running, Role: LEADER
I20260806 20:16:27.683029 1361 consensus_queue.cc:260] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.684269 1361 raft_consensus.cc:3037] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:27.686394 1239 raft_consensus.cc:504] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.688087 30868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee7927d05584c999f6e0a2b753336f1" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.688531 1239 leader_election.cc:288] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:27.689160 30788 raft_consensus.cc:1226] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 58f2f6256ee14cb38a1dbcbf5bb590a0 for earlier term 6. Current term is 7. Ops: []
I20260806 20:16:27.689698 1489 raft_consensus.cc:2781] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 20:16:27.690121 1489 raft_consensus.cc:482] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:27.690452 1489 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:27.691577 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26a3d1fc21e040c09103cfcf9e539659" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.692375 30865 raft_consensus.cc:2379] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 5: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:27.693267 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88608755c61d42ec8ccc0c3c8e536bd6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.696661 1293 consensus_queue.cc:1046] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 20:16:27.698113 987 raft_consensus.cc:3032] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:27.698511 987 raft_consensus.cc:729] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.647791 1400 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.699355 987 consensus_queue.cc:260] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.700326 987 raft_consensus.cc:3037] T 1ee7927d05584c999f6e0a2b753336f1 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
I20260806 20:16:27.700680 1305 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.701138 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e68e22f574e679ad1e327caa342f6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.701807 30864 raft_consensus.cc:2454] T b32e68e22f574e679ad1e327caa342f6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 7.
I20260806 20:16:27.701844 30869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.702549 30869 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9.
I20260806 20:16:27.681355 1479 raft_consensus.cc:3037] T f53758bb873349479f86e0abf34c738b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 10 FOLLOWER]: Advancing to term 11
I20260806 20:16:27.703222 30863 raft_consensus.cc:1261] T 8ec60d6f5b6c486b80781ad3cb85a06b P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:16:27.703579 1305 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:27.663494 961 raft_consensus.cc:2726] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 20:16:27.704953 30870 raft_consensus.cc:2396] T 88608755c61d42ec8ccc0c3c8e536bd6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 20:16:27.700717 1400 leader_election.cc:288] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735), 58f2f6256ee14cb38a1dbcbf5bb590a0 (127.28.186.196:35519)
I20260806 20:16:27.695642 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78bef7be8f95499c84c37311213c95b6" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.698006 986 raft_consensus.cc:504] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.712414 30867 raft_consensus.cc:2454] T cc6e5d61a5694983a73c242e91257d86 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:27.704169 30835 leader_election.cc:302] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.709782 986 leader_election.cc:288] T b7665c0d8832432f8948e82d823498fd P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 9 election: Requested vote from peers d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:27.727523 31422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" is_pre_election: true
I20260806 20:16:27.728180 31422 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:27.733284 1390 leader_election.cc:288] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735)
I20260806 20:16:27.733708 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.722528 1293 raft_consensus.cc:2781] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:27.734388 30861 raft_consensus.cc:2454] T 761e8328b9d247fcb1d00f0aeff373f0 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 9.
I20260806 20:16:27.735118 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7665c0d8832432f8948e82d823498fd" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.735565 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:27.733503 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0"
I20260806 20:16:27.739674 30787 raft_consensus.cc:2379] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 in current term 9: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:27.740517 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:27.746727 30754 leader_election.cc:302] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f2f6256ee14cb38a1dbcbf5bb590a0, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.754472 30861 raft_consensus.cc:1261] T 9f32336173df4611829be37d396e86ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 20:16:27.757081 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79dfdb4a0e9a40d39c83f2c1fd1e777b" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:27.724964 1475 consensus_queue.cc:1046] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260806 20:16:27.738950 1293 raft_consensus.cc:482] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:27.762763 30790 raft_consensus.cc:1226] T 672b273ef36e4c6eb6873cc61471cb53 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Rejecting Update request from peer 42cabca48db7403b8523a3382a046043 for earlier term 6. Current term is 7. Ops: []
W20260806 20:16:27.764041 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 208812us
I20260806 20:16:27.765618 30835 leader_election.cc:302] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.770335 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b060012888ff4d76b254b43d1f1959bb" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.770532 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412dab7986704f48be75b1fdbdccc9fa" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:27.755614 1495 raft_consensus.cc:2781] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 20:16:27.773101 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4430b33a44474cbfbc682c4bef43e2f5" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:27.775243 30787 raft_consensus.cc:3037] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:27.762346 1293 raft_consensus.cc:3037] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:27.771593 961 raft_consensus.cc:2781] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 20:16:27.778139 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:27.778841 30783 raft_consensus.cc:1261] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 20:16:27.782349 30866 raft_consensus.cc:2454] T 15c9bdefd8144ce787ecb60a54ea1450 P 58d59beb597f4ed99d84d102ddb41ff5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 6.
I20260806 20:16:27.777230 1495 raft_consensus.cc:482] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:27.795368 30791 raft_consensus.cc:2354] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42cabca48db7403b8523a3382a046043 for earlier term 8. Current term is 9.
I20260806 20:16:27.790330 30787 raft_consensus.cc:2454] T 4430b33a44474cbfbc682c4bef43e2f5 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d59beb597f4ed99d84d102ddb41ff5 in term 10.
I20260806 20:16:27.797154 30676 leader_election.cc:302] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters: 58d59beb597f4ed99d84d102ddb41ff5
W20260806 20:16:27.793771 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 248069us
I20260806 20:16:27.802170 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2a287c4de469eb5365fa7cdc6b409" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.807508 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c9bdefd8144ce787ecb60a54ea1450" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:27.812367 30756 leader_election.cc:302] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:27.814505 1390 consensus_queue.cc:1035] T 8a0db3302c934290a93017bbfb29db44 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 20:16:27.815542 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c1793b412c4608b1e004525a6dd887" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
W20260806 20:16:27.816948 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9f32336173df4611829be37d396e86ba, TS 42cabca48db7403b8523a3382a046043 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:27.817276 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9f32336173df4611829be37d396e86ba, TS 42cabca48db7403b8523a3382a046043 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.817930 30706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043" is_pre_election: true
I20260806 20:16:27.818603 30706 raft_consensus.cc:2454] T a439c5debb41440c913a50da91e6467f P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 8.
I20260806 20:16:27.797295 1495 raft_consensus.cc:3037] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 4 FOLLOWER]: Advancing to term 5
I20260806 20:16:27.820295 1407 raft_consensus.cc:482] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:16:27.820807 1407 raft_consensus.cc:504] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.817387 30712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "42cabca48db7403b8523a3382a046043"
I20260806 20:16:27.822561 30786 raft_consensus.cc:595] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Received request to transfer leadership to TS 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:27.822932 30786 raft_consensus.cc:603] T 0911118ae8e247aca188955eb02c8728 P d4181d4d907f477f9feddc44ba3b9207 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 20:16:27.826894 31414 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:27.807376 961 raft_consensus.cc:686] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [term 5 LEADER]: Becoming Leader. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.829646 1309 consensus_queue.cc:1046] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 20:16:27.829959 1499 raft_consensus.cc:2726] T 7cb9334179bc40b1b3868fc4773af362 P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:27.832850 1407 leader_election.cc:288] T 95c5938e508b4075b878ae1f5ea726d6 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:27.835448 1407 raft_consensus.cc:3032] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 20:16:27.839772 1407 raft_consensus.cc:729] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58d59beb597f4ed99d84d102ddb41ff5, State: Running, Role: LEADER
I20260806 20:16:27.844514 1407 consensus_queue.cc:260] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.813993 30756 leader_election.cc:302] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
I20260806 20:16:27.846949 1407 raft_consensus.cc:3037] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 7 FOLLOWER]: Advancing to term 8
W20260806 20:16:27.837316 30655 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 33e2a287c4de469eb5365fa7cdc6b409, TS 42cabca48db7403b8523a3382a046043 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
W20260806 20:16:27.853673 30655 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 33e2a287c4de469eb5365fa7cdc6b409, TS 42cabca48db7403b8523a3382a046043 -> TS 58f2f6256ee14cb38a1dbcbf5bb590a0 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.857084 30756 leader_election.cc:302] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.858047 1361 consensus_queue.cc:1035] T 942abb68076e47a0a24672bb92649892 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:16:27.860375 30791 raft_consensus.cc:1226] T cca4a581dd9f4cff973793835675b6c3 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:27.861574 1309 raft_consensus.cc:482] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d4181d4d907f477f9feddc44ba3b9207)
I20260806 20:16:27.862272 1309 raft_consensus.cc:504] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.864607 1309 leader_election.cc:288] T 0911118ae8e247aca188955eb02c8728 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:27.868716 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca4a581dd9f4cff973793835675b6c3" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.869306 30861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba15abedb154a46b9f13784776fbeae" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.870097 1338 raft_consensus.cc:3032] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 20:16:27.870550 1338 raft_consensus.cc:729] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58f2f6256ee14cb38a1dbcbf5bb590a0, State: Running, Role: LEADER
I20260806 20:16:27.870288 30867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1066045a99d4a1da0fff49e2b8bbe30" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.871520 1338 consensus_queue.cc:260] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:27.872164 961 consensus_queue.cc:237] T 26a3d1fc21e040c09103cfcf9e539659 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.873018 1338 raft_consensus.cc:3037] T e0c1793b412c4608b1e004525a6dd887 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 6 FOLLOWER]: Advancing to term 7
I20260806 20:16:27.848347 1496 raft_consensus.cc:2726] T e1485b55849242feae0e8b2b3b28307f P d4181d4d907f477f9feddc44ba3b9207 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 20:16:27.881526 30790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6868aeec902a4311a782f4ef5ab064f8" candidate_uuid: "42cabca48db7403b8523a3382a046043" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:27.882359 30790 raft_consensus.cc:3037] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:27.883081 31414 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:16:27.885381 1499 consensus_queue.cc:1035] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 20:16:27.898039 30756 leader_election.cc:302] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58f2f6256ee14cb38a1dbcbf5bb590a0
W20260806 20:16:27.906522 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 109658us
I20260806 20:16:27.923619 30756 leader_election.cc:302] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42cabca48db7403b8523a3382a046043, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.932631 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cdb10a40ac4d0bbcfc4ed368b6dc43" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
W20260806 20:16:27.936165 30835 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 117247us
I20260806 20:16:27.939812 30861 raft_consensus.cc:1261] T 8a0db3302c934290a93017bbfb29db44 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260806 20:16:27.939615 30757 leader_election.cc:334] T 1ee7927d05584c999f6e0a2b753336f1 P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 1.729s (SENT)
I20260806 20:16:27.907558 1499 raft_consensus.cc:2781] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 20:16:27.969059 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:27.938633 30835 leader_election.cc:302] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d59beb597f4ed99d84d102ddb41ff5, d4181d4d907f477f9feddc44ba3b9207; no voters:
I20260806 20:16:27.924618 1019 raft_consensus.cc:2726] T e0c1793b412c4608b1e004525a6dd887 P d4181d4d907f477f9feddc44ba3b9207 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 20:16:27.971704 30870 raft_consensus.cc:2454] T 48c353d311a64e628bfc1af985e1d8ba P 58d59beb597f4ed99d84d102ddb41ff5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4181d4d907f477f9feddc44ba3b9207 in term 4.
I20260806 20:16:27.981027 1309 raft_consensus.cc:2781] T 4430b33a44474cbfbc682c4bef43e2f5 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 20:16:27.986917 30865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
I20260806 20:16:27.925158 1501 raft_consensus.cc:2781] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 20:16:27.990729 1498 log.cc:924] Time spent T e0607668d3924d5cba6acd89b2cdc25f P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 20:16:27.991675 1293 raft_consensus.cc:504] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:27.994796 1489 raft_consensus.cc:504] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 } } peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } }
I20260806 20:16:28.006223 30865 raft_consensus.cc:2379] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 in current term 9: Already voted for candidate 58d59beb597f4ed99d84d102ddb41ff5 in this term.
I20260806 20:16:27.937774 1500 raft_consensus.cc:2781] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 20:16:27.905674 1496 consensus_queue.cc:1035] T 9f32336173df4611829be37d396e86ba P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260806 20:16:27.996903 1489 leader_election.cc:288] T aa7cfe759ca64ca89a0529c2d3da5bf7 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 10 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:27.974089 1499 raft_consensus.cc:482] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:28.006644 30790 raft_consensus.cc:2454] T 6868aeec902a4311a782f4ef5ab064f8 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42cabca48db7403b8523a3382a046043 in term 9.
I20260806 20:16:27.970364 987 consensus_queue.cc:1035] T a439c5debb41440c913a50da91e6467f P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260806 20:16:28.012570 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c5938e508b4075b878ae1f5ea726d6" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207" is_pre_election: true
I20260806 20:16:28.124758 30787 raft_consensus.cc:2396] T 95c5938e508b4075b878ae1f5ea726d6 P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260806 20:16:28.009568 1500 raft_consensus.cc:482] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:16:28.128046 1500 raft_consensus.cc:3037] T a439c5debb41440c913a50da91e6467f P d4181d4d907f477f9feddc44ba3b9207 [term 8 FOLLOWER]: Advancing to term 9
I20260806 20:16:28.127955 30706 raft_consensus.cc:1261] T 79dfdb4a0e9a40d39c83f2c1fd1e777b P 42cabca48db7403b8523a3382a046043 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260806 20:16:28.032279 30757 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.195:41735, user_credentials={real_user=slave}} blocked reactor thread for 92688.6us
I20260806 20:16:28.063987 31422 raft_consensus.cc:1261] T a1066045a99d4a1da0fff49e2b8bbe30 P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 8 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 20:16:28.031131 30870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a439c5debb41440c913a50da91e6467f" candidate_uuid: "d4181d4d907f477f9feddc44ba3b9207" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58d59beb597f4ed99d84d102ddb41ff5" is_pre_election: true
I20260806 20:16:28.061616 30789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7cfe759ca64ca89a0529c2d3da5bf7" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:28.137995 30789 raft_consensus.cc:3037] T aa7cfe759ca64ca89a0529c2d3da5bf7 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:28.064958 30786 raft_consensus.cc:1226] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [term 8 LEADER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 7. Current term is 8. Ops: []
I20260806 20:16:28.138056 30791 raft_consensus.cc:1226] T 7ba15abedb154a46b9f13784776fbeae P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Rejecting Update request from peer 58d59beb597f4ed99d84d102ddb41ff5 for earlier term 4. Current term is 5. Ops: []
I20260806 20:16:28.141027 30787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b0fcea699eb4ec884b7451428998012" candidate_uuid: "58d59beb597f4ed99d84d102ddb41ff5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
I20260806 20:16:28.141947 30787 raft_consensus.cc:2379] T 2b0fcea699eb4ec884b7451428998012 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d59beb597f4ed99d84d102ddb41ff5 in current term 9: Already voted for candidate d4181d4d907f477f9feddc44ba3b9207 in this term.
I20260806 20:16:27.997367 1501 raft_consensus.cc:482] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
W20260806 20:16:28.146721 1503 log.cc:924] Time spent T 0f3393f94b2649378cb19b57fecfb21e P 58f2f6256ee14cb38a1dbcbf5bb590a0: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
W20260806 20:16:28.137506 30676 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.186.194:42029, user_credentials={real_user=slave}} blocked reactor thread for 167954us
I20260806 20:16:28.156313 1497 raft_consensus.cc:482] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58d59beb597f4ed99d84d102ddb41ff5)
I20260806 20:16:28.157033 1390 consensus_queue.cc:1035] T a1066045a99d4a1da0fff49e2b8bbe30 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 20:16:28.156932 1497 raft_consensus.cc:504] T a5e69bf76cc845c8822b4cfab14eac47 P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5" member_type: VOTER last_known_addr { host: "127.28.186.195" port: 41735 } } peers { permanent_uuid: "58f2f6256ee14cb38a1dbcbf5bb590a0" member_type: VOTER last_known_addr { host: "127.28.186.196" port: 35519 } attrs { promote: false } }
I20260806 20:16:28.137266 1499 raft_consensus.cc:3037] T f0cdb10a40ac4d0bbcfc4ed368b6dc43 P d4181d4d907f477f9feddc44ba3b9207 [term 9 FOLLOWER]: Advancing to term 10
I20260806 20:16:28.165417 30867 raft_consensus.cc:2396] T a1066045a99d4a1da0fff49e2b8bbe30 P 58d59beb597f4ed99d84d102ddb41ff5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4181d4d907f477f9feddc44ba3b9207 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260806 20:16:28.166551 1337 consensus_queue.cc:1046] T 7ba15abedb154a46b9f13784776fbeae P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 20:16:28.174547 30706 raft_consensus.cc:1261] T 761e8328b9d247fcb1d00f0aeff373f0 P 42cabca48db7403b8523a3382a046043 [term 9 FOLLOWER]: Refusing update from remote peer d4181d4d907f477f9feddc44ba3b9207: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260806 20:16:28.064443 1293 leader_election.cc:288] T 2b0fcea699eb4ec884b7451428998012 P 58d59beb597f4ed99d84d102ddb41ff5 [CANDIDATE]: Term 9 election: Requested vote from peers 42cabca48db7403b8523a3382a046043 (127.28.186.193:42937), d4181d4d907f477f9feddc44ba3b9207 (127.28.186.194:42029)
I20260806 20:16:28.031471 1480 consensus_queue.cc:1046] T cca4a581dd9f4cff973793835675b6c3 P 58d59beb597f4ed99d84d102ddb41ff5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.135s
I20260806 20:16:28.183107 31422 raft_consensus.cc:1261] T 7c541d91c0c4487c9a76c6fc6f81a95a P 58f2f6256ee14cb38a1dbcbf5bb590a0 [term 7 FOLLOWER]: Refusing update from remote peer 58d59beb597f4ed99d84d102ddb41ff5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 20:16:28.182212 30791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48c353d311a64e628bfc1af985e1d8ba"
dest_uuid: "d4181d4d907f477f9feddc44ba3b9207"
mode: GRACEFUL
new_leader_uuid: "58d59beb597f4ed99d84d102ddb41ff5"
from {username='slave'} at 127.0.0.1:47144
I20260806 20:16:28.151261 1501 raft_consensus.cc:3037] T 7ba15abedb154a46b9f13784776fbeae P 42cabca48db7403b8523a3382a046043 [term 4 FOLLOWER]: Advancing to term 5
W20260806 20:16:28.185655 30757 leader_election.cc:334] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58d59beb597f4ed99d84d102ddb41ff5 (127.28.186.195:41735): Timed out: RequestConsensusVote RPC to 127.28.186.195:41735 timed out after 2.162s (SENT)
I20260806 20:16:28.188529 30757 leader_election.cc:302] T dc53de31500c4908b8ec8ee3a0f7498c P d4181d4d907f477f9feddc44ba3b9207 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d4181d4d907f477f9feddc44ba3b9207; no voters: 42cabca48db7403b8523a3382a046043, 58d59beb597f4ed99d84d102ddb41ff5
I20260806 20:16:28.180413 1390 consensus_queue.cc:1035] T 761e8328b9d247fcb1d00f0aeff373f0 P d4181d4d907f477f9feddc44ba3b9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42cabca48db7403b8523a3382a046043" member_type: VOTER last_known_addr { host: "127.28.186.193" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260806 20:16:28.083437 1495 raft_consensus.cc:504] T 48c353d311a64e628bfc1af985e1d8ba P d4181d4d907f477f9feddc44ba3b9207 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d4181d4d907f477f9feddc44ba3b9207" member_type: VOTER last_known_addr { host: "127.28.186.194" port: 42029 } } peers { permanent_uuid: "58d59beb597f4ed99d84d102ddb41ff5"